And, I'm sorry... I should have mentioned that I'm using the Due. It is a lot more RAM and DMA so I'd really want to not only do 400Khz and use large buffers but also use DMA to send and receive. On the Due with it's 100KB of RAM it would be no problem to have 128 byte buffers * 5 which is still less than a kilobyte and thus less than 1% of the RAM.
The TWI library on the Due is already a bit different from the 1.0.3 AVR version. Should I have left this discussion in the Due forums?