I'm using an SD card to store sensor data in a CSV. The device is designed to transfer the data from the SD to a tablet via a usb connection, but there is a max buffer of 4 kb and the data file averages 300 kb. Does anyone have any experience parsing data files from SD in a similar application?
Excel reads csv.
Mathcad reads csv.
Any program langage parse csv.
but there is a max buffer of 4 kb
Where? The size of the buffer really should be immaterial, since you can't have a 4kb buffer on the Arduino (not on most of them, anyway).
The program that owns the buffer should be emptying it far faster than you can fill it over a serial connection.