Formatting Data for Data Logging

Other Hardware Storage
1

I'ts been a while since I've worked with Arduino but I want to get back into it. I'm working on a data logger for my motorcycle and I wan't this data continuously written to an SD card so that I can review my performance. I have 17 channels, each channel will have 1-24 bits of data to log, and the data will be logged at 10Hz. I'm going to be close to the limit of the write speed to the SD card so I would like to know how I can format the raw data in the most minimal form.

If I separate each channel with a comma or some other ASCII character, thats 8 bits for the character, there are 17 channels so I need 16 separators, and the data is being written 10 times a second. Thats 1280 bps that need to be written.

If i store the data in DEC form, is there extra data being transmitted in there to tell the SD card that each number is supposed to be DEC?

I think i really want to store the data in binary but I'm curious if I have an 8 bit block sensor data and the sensor value is only say DEC 7, will the data be stored on the SD card as 111 or 00000111? If its the latter that would be awesome as I could then post process the data based on number of bits in each channel. Example: 3 channels, 1st has 2 bits, 2nd has 4 bits, 3rd has 2 bits. If my data comes back as 10010111 Then I know channel 1 is reading 2, Channel 2 is reading 10, and channel 3 is reading 3.

I hope this makes sense and I thank you all for helping out!

2

I'm going to be close to the limit of the write speed to the SD card

Your average data rate is no problem. If each data item took 10 bytes, 17 items at 10 Hz would be less than 2 KB/sec.

The problem could be some lost points due to occasional long write latency. The SD spec allows occasional write latencies of over 100 ms. The SD spec allows latencies of up to 250 ms as long as average transfer rates are maintained.

The only sure way to log data at rates faster than about 5 Hz is to use a separate thread to read sensors or read sensors in an interrupt routine. Data is then buffered for a thread that write data to the SD card.

I have posted examples of both solutions. The most general uses a small RTOS. See the nilSdLogger example in NilRTOS20130720.zip here Google Code Archive - Long-term storage for Google Code Project Hosting..