High speed data logger solutions

Other Hardware Storage
1

I plan to use Arduino Mega 2560 to realize some real-time control (1ms timer interruption) of mechatronic devices.
And I now have a trouble with saving control data (aournd 10 items) at high speed.

e.g.,

  1. I tried SD card to store data, but I found the consuming time of saving data is long (maybe around 10ms?)

  2. I also tried directly transfer data to PC through serial port, but writing around 10 items at one control loop consumes almost 0.03ms...... 30 times of the control loop!

Is there any solution for high speed data save, at least 5ms?
e.g.,
control loop: 1ms loop
data logger: 5ms loop
so, what I got is data of 0ms, 5ms, 10ms, 15ms, 20ms..... at discrete-time.

2

ttcaixp:
I plan to use Arduino Mega 2560 to realize some real-time control (1ms timer interruption) of mechatronic devices.
And I now have a trouble with saving control data (aournd 10 items) at high speed.

e.g.,

  1. I tried SD card to store data, but I found the consuming time of saving data is long (maybe around 10ms?)

  2. I also tried directly transfer data to PC through serial port, but writing around 10 items at one control loop consumes almost 0.03ms...... 30 times of the control loop!

Is there any solution for high speed data save, at least 5ms?
e.g.,
control loop: 1ms loop
data logger: 5ms loop
so, what I got is data of 0ms, 5ms, 10ms, 15ms, 20ms..... at discrete-time.

Buffer the data in multiple larger (512byte) blocks. When one block is full write it.

Use the other block to save the new samples while the first block is being written.

If you have 10 items (lets say 2 bytes each) so 20bytes every 1 ms. with a 512byte buffer you can buffer 25 samples (25ms). As long as the SDcard write takes less than 25ms you are golden.

With the Mega you have 8k of RAM, so if you allocate 4k for your buffer, that becomes 8 512byte buffers. Most SDcard writes will complete within 25ms, as long as the average is less than 25ms, you can use the additional buffers when it takes longer.

Chuck.

3

Thanks, chucktodd.

I will try your method.

By the way, I am also considering adding external SPI SRAM to store data.
And after the task completed, send all data to PC or SD card at one time.
Do you know how fast writing to a SPI SRAM?

4

ttcaixp:
Thanks, chucktodd.

I will try your method.

By the way, I am also considering adding external SPI SRAM to store data.
And after the task completed, send all data to PC or SD card at one time.
Do you know how fast writing to a SPI SRAM?

With the Mega the SPI clock can be set to 8mhz, I have never actually it at 8mhz, I just use the default 4mhz, so with a 23LC512 (64kByte) SRAM, a write sequence takes 1 byte cmd, 2 bytes address, + data.

The standard SPI library can transfer as big a buffer as will fit in RAM, so your 512 byte buffers would be easy to transfer. If you can do your Sampling during an interrupt, it would happen simultaneously.

if you are using a 4mhz SPI SCK, for a 512byte data block, that is 515bytes * 8 SCK(bits)=4120 clocks at 4mhz or 1.03ms. You have to add a few microseconds for program overhead.

Chuck.

5

Here are two other alternatives. Neither of those alternatives are supported by default in the Arduino IDE and it's probably quite a bit of work to re-implement your sketch for one of them. If you want to give alternative 1 a try I can write a sketch with the basic functionality you need.

  1. Use an RTOS with two threads; one for the control loop and one background thread for the logging. For example you can use my RTOS Simba. It supports multi threading and has an SD driver + FAT16 file system. =)

  2. Use an asynchronous API to write to the SD card. Start writing samples to the SD card and check if the write has completed in the control loop, without blocking it. I havn't found a library that supports this, but there must be one out there. =)