Why the boat loader ?

A TTL USB adapter and a bootloaded ATMEGA is a simple recipe. I am not sure if you can program any TTL USB adapter to use ICSP header. Some probably could. If you program a small chip with an ICSP programmer or another Arduino Dev board, you will erase the original bootloader (if the small chip had one) and your code will take the entire Flash as needed. Remember, Arduino is not a project that is trying to cram a project into the smallest and cheapest chips, like an engineering product.