KK959, can you tell what the project is ? What are the baudrates, and how many bytes is the block of data that is received ?
For example with a high baudrate of 115200, it takes 11 ms to transfer 128 bytes. That is not a lot, but during those 11 ms there is enough time to read data from the buffer.
With 9600 baud it takes 133 ms to transfer 128 bytes.
Large buffers are not a waste of memory if you don't need that memory 
I can turned that around: you paid for the memory, it is a waste if you don't use it 
If you really need large buffers, then you can upgrade to an Arduino Due, and increase the Serial buffer size.