I didn't try Optiboot. I briefly looked at it, but the original one looked easier to figure out, plus I had the Sanguino example to start from.
I think I've got it figured out - I looked at the Sparkfun Pro Mini board settings for 8MHz with a '328, as they were using 57600 baud and I know that works as I've used those boards in other projects.
Anyhow, the trick seemed to be that they were using the 'double speed' option in the baud rate, so I copied what they were doing for the '328 and tried it on the '1284p and it worked.
In the makefile, my cflags are set to:
CFLAGS += '-DMAX_TIME_COUNT=F_CPU>>4' '-DNUM_LED_FLASHES=1' -DBAUD_RATE=57600 -DDOUBLE_SPEED
and in the ATmegaBOOT_168.c source file, In the section for initializing the UART(s) I changed the __AVR_ATmega1284P__ section to this:
UCSR0A = (1<<U2X0); //Double speed mode USART0
UBRR0L = (uint8_t)(F_CPU/(BAUD_RATE*8L)-1);
UBRR0H = (F_CPU/(BAUD_RATE*8L)-1) >> 8;
UCSR0A = 0x00;
UBRR0L = (uint8_t)(F_CPU/(BAUD_RATE*16L)-1);
UBRR0H = (F_CPU/(BAUD_RATE*16L)-1) >> 8;
UCSR0C = 0x06;
UCSR0B = _BV(TXEN0)|_BV(RXEN0);
The only thing that I've omitted is there was originally settings for both UART0 and UART1 and I'm only using UART0 so I've dropped the stuff for UART1 in the above code.
57600 is nice and snappy, compared to 19200.