I've been uploading to my 1284P via ICSP due to this problem, even just a 23k sketch failed every time via the bootloader and serial.
I've just re-burnt the bootloader and tried the same sketch 3 times and it failed.
I added the 10k series resistor on RX0 and tried again and it worked!
Tried it 6 times and it uploaded 100%.
Removed the 10k resistor and it failed.
Replaced the resistor and it worked 3 times in a row again.
I'll now try re-burning the fuses with the 0xF7 low bit and report back.
EDIT: Used Nick's Bootloader Programmer (for the first time) to re-burn the bootloader and set the fuses (I changed Low Fuse from 0xFF to 0xF7 in his sketch) and retried the upload with no resistor. It worked! Now I just need a big sketch to verify it really.
The most profound thanks to you guys for working together to track down and find a solution to this problem. Ye, you can upload via ICSP, but when your board is in a device that means you have to take everything off the SPI bus. On the board I just had made I actually included a 4 pin DIP switch so I could separate everything else form the ICSP header for any code updates that needed to be programmed. I also have a serial header available as well.
Right, now it appears that the 644P family is in the sketch to be identified but there is no entry for a bootloader or fuses setup. Does anyone know how we might go about adding 644P support?