Arduino ATmega1284p bootloader

You need some decoupling caps on the VCC/AVCC pins.
You also need to connect FTDI's input pin CTS to Gnd so it knows the chips is waiting for data.