Hi Wiz,
I believe I have just been using the Sanguino bootloader.
You are correct, the extended core files from Mark do not include a bootloader for the Duino-644P. I did a quick google search and found an adafruit forum topic which said to just use the Sanguino bootloder.
You think the problem exists in the bootloader itself?
You should try the new core files without the bootloader, adding an extry into the boards.txt file, to use whatever ISP programmer you use instead - and see if that has the same problem.
This is one I use, you may need to change it to suit the ISP programmer you use if it is different.
atmega644p.name=[usbtinyisp]ATmega644P-Duino
atmega644p.upload.using=usbtinyisp
atmega644p.upload.maximum_size=63488
# Ext. Crystal Osc 8Mhz+, Start-up time 16K CK + 65 ms; [CKSEL=1111 SUT=11]
# Brown-out detection level at VCC=4.3 V; [BODLEVEL=100]
# Serial program downloading (SPI) enabled; [SPIEN=0]
# Boot Flash section size=512 words Boot start address=$7E00; [BOOTSZ 11]
# Boot Reset vector Disabled [BOOTRST=1]
atmega644p.bootloader.low_fuses=0xFF
atmega644p.bootloader.high_fuses=0xDF
atmega644p.bootloader.extended_fuses=0xFF
atmega644p.bootloader.path=empty
atmega644p.bootloader.file=empty644p.hex
atmega644p.bootloader.unlock_bits=0x3F
atmega644p.bootloader.lock_bits=0x0F
atmega644p.build.mcu=atmega644p
atmega644p.build.f_cpu=16000000L
atmega644p.build.core=arduino
The blank empty644P.hex file specified was created using this method, Message # 15 from Coding Badly:
Make sure you add this to the boards.txt inside the new core file directory (arduino-extras).
So basically, connect the chip to the ISP, select this new board, write the bootloader (which is just a blank file), and then program it and see what happens.
Let me know the results.