As pito figured out, set that low fuse to full-swing (0xF7) in the programmer sketch:
// ATmega1284P family
{ { 0x1E, 0x97, 0x05 }, "ATmega1284P", 128 * kb, 1 * kb,
optiboot_atmega1284p_hex,
0x1FC00, // start address
sizeof optiboot_atmega1284p_hex,
256, // page size (for committing)
0xD7, // fuse low byte: external clock, 0xD7 = FULL SWING
0xDE, // fuse high byte: SPI enable, boot into bootloader, 1024 byte bootloader
0xFD, // fuse extended byte: brown-out detection at 2.7V
0x2F }, // lock bits: SPM is not allowed to write to the Boot Loader section.