Data logger - time taken to write a line to an SD card.

Hi,

How long should it take to write about 100 characters to a SD card?

I have a project that (among other things) reads position information from a GPS module, and logs data to an SD card.

The GPS data is arriving at the serial port at a rate of 57600, experimentation indicates that I need to read and process the data at least once every 10-12 ms to ensure that serial buffer doesn't get filled up and data get lost. The project also needs to log data to a file on the SD card, data is logged once a second, and each line of data is about 100 characters long. The project also process other stuff, lots of navigational maths, distance/track/course/windspeed/ETA/Heartrate/power etc etc.

I've experimented with using millis() before and after each line is written to the logger, and have found that each write seem to take between 15 and 25 milliseconds. This seems to be the single longest individual step that the application does, and the result is that occasionally the GPS data gets lost because the port has overflown whilst waiting for the next read.

So, back to my original question, how long should it take to write a single line of about 100 characters to an SD card. 15-25 milliseconds seems a bit long to me, but maybe that's normal?

I'm not looking for a specific coding fix (the project is too long and complicated to post the whole thing here) just some general advice. I'm using an arduino mega, the SD card is initialised at SPI_FULL_SPEED, and all the testing I've done indicates that the single most time consuming action the project performs (by far) is writing data to the SD card.

Cheers

Isn't that fast for a GPS? Can't you slow it down to 4800 baud or something?

57600 is the default speed for the GPS, I could probably slow it down, but one of my objectives is to accurately track my position at a rate of one fix per second - slowing down the GPS data rate would have a negative impact on this.

What I really want to do is try to speed up the writing to the SD card. I know this is the most time-consuming thing my application does, but I don't know if it's taking longer than it ought to. Is 15-25 milliseconds for 100 characters about right? Or should it be possible to speed up the process? I know writing to a hard disk takes time, but I though writing to a solid-state device ought to be done in the blinking of an eye!

Cheers

Writing to EEPROM is slower than reading. Are you closing the file each time? That could take time as disk directory structures are updated.

Look at fat16lib's fastlogger: Google Code Archive - Long-term storage for Google Code Project Hosting.

SD cards read/write in blocks of data, not single bytes.

The SD library uses a 512 byte cache. If you write a 100 bytes then sync() ( or close the file ), this causes the unused cache to be filled from the SD ( read operation, to preserve data already on SD block ). Then the entire 512 byte block is written to the SD.

Use a buffer that is a factor of 512 to write out contigous blocks. fastLogger uses a 'BufferedWriter' class to work on a 64 byte block. This allows the cache blocks to be filled without requiring reads.

Look at fat16lib's fastlogger

It looks good, I'll do some research on this.

The SD library uses a 512 byte cache. If you write a 100 bytes then sync() ( or close the file ), this causes the unused cache to be filled from the SD ( read operation, to preserve data already on SD block ). Then the entire 512 byte block is written to the SD.

I think you're saying that writing 100 bytes will take just as long (or longer) to write than 512 bytes. That's interesting, I think it may also explain why sometimes the last couple of lines of logged data appear to have been lost when the arduino is suddenly switched off.

thanks

Yeah a small write could easily be longer than a full block write.

I'd also imagine that stopping the Arduino while its in the middle of manipulating the SD card could possibly cause file system errors, not just missing data. I'm only making assumptions, but if its bad for the SD, a button to 'shutdown' the system may be worth it.