Yeesh, nevermind, I found it.... stupid programming mistake.
When I read from the serial port I looped through the buffer and found all my 3-byte messages and then copied any remaining bytes to the beginning of the buffer to be processed on the next go-round. When I read from the serial port the next time I just offset the read() after the old bytes... almost... I was accidentally offsetting by both the current extra bytes and the extra bytes from the previous read...
I now lose 1 in several hundred million bytes at 195Kbyte/sec(2Mbaud)... which is very reasonable for an asynchronous serial port with no flow-control.
Thanks for letting me know this problem wasn't normal... It's hard to trust a platform you aren't familiar with.
Now to get my 3Mbaud connection working this well... I still get pretty high error rates (1 in 50000 bytes goes missing). I'm not certain if this is because:
- I'm pushing the limits of the FTDI chip. It maxes out at 3Mbaud
- I'm pushing the limits of the Amtel328... it maxes out at 4Mbaud @ 16Mhz
- My frequency divider may be crap; it's made from stuff I had laying around. I need a real oscilloscope...