If you download your program via the USB/Serial port, there is a slight delay after power up while the bootloader checks to see if another program will be downloaded, and if not, your program is started.
If you acquire another tool, such as this
http://www.mdfly.com/index.php?main_page=product_info&cPath=5&products_id=415
than you can download your program via the ICSP header and it will start immediately on power up.
You will have to download the bootloader again if you then want to go back to using the serial port.
The code will not run from EEPROM, that is there for the running program to store things, that the running program can access again after a reset/powerloss or some other time in the future.