@PaulS: you're right. When my direct streaming design cannot handly timing issues, I do need to switch to this 64byte-sample approach. However, I will need to redisign my protocol. That's to say, it's only sending full information today.
Knowing the speed of ethernet SPI, I could go down one step with my serial speed. Does anyone knows about the speeds?
Update: default for SPI is 4 Mhz see SPI - Arduino Reference. I wonder why my code accumulate the rx buffer. Maybe the serial3.available() takes as much time as serial3.read(). Any ideas?