High speed Data logging project

Hi guys,

I am new to the arduino world and it has been a long time in general since I had to program myself.

So my current problem is, that I have to measure 3 ADC 10 bit values all 100us and save these data to an sd card. the measurement time is 2 seconds worst case. Do you have any suggestions or is there already an adaptable sketch for doing so ?

Thank you

2,1 or even .2 seconds is not all that high speed. Plenty of time to write data to the SD card.

Not easy. IIRC, the Uno takes 100uS to read an analog pin and the single ADC is shared between the pins and readings may be influenced by the previous one, so there's often a need to read twice get an accurate result, so it's already six times too slow.

You're going to end up with ~half a meg of data to write out too, and I suspect that the SD card will not be able to keep up.

You may want to get something a lot faster than an Uno, possibly some external ADCs and ethernet to send the data to a server.

So the Uno only has one ADC shared between all ADC pins ?
What do you mean by accurate ? How much is it off due to this ?

I have an class 10 SD card, I think it is fast enough.

I have read something about a data logger that is up to 100ksps so why should this timing be a problem?

If you read the ADC, the reading you get may influence the next read. If that happens, you need to read the pin again.

As to the SD card , simple enough to try it to see if it is fast enough.

A mistake that many people make is to open the file, write a bit of data, then close it again, which slows down the process by a factor of ten or more and increases the error rate.

Open the file once in setup(), then in loop() read data and write to the card. Close the file only when data collection is completely finished.

Most microprocessors have only one ADC, if any.

Writing to an SD card takes almost NO time while you are filling the 512 byte buffer, then it takes lots of time to physically write the buffer and get ready to refill it.

1 Like

So do you have an example sketch for me on how to program those buffer ? Honestly I have no idea how to do it

Every time your program writes to or reads from an SD card, you are using the buffer. Your program does not see all the rest of the processing that goes on to handle the physical writing and reading.

To get a realistic timing test, you need to write enough bytes to fill perhaps, a dozen 512 byte buffers and average that time.

So to do so, I make an (for example) analogread, save it in a string and read the next, separated by a ";" and so on until I reach that 512 byte and then write it to the SD card ? How do I detect how big that string is ?

It does not matter where the data comes from, if you want to time the writing process.
Fill a big file with whatever you like and time the duration it takes.

You want to save 3 values (6 bytes) all 100 us, makes 6 *10 * 1000 * 2 = 120000 byte,
try to bring 120000 bytes of constant data to the SD card.
The time that you use below 2 seconds, is the time you have left for processing or gathering data.

I know, but in best case I could use the data and structure for my measurement and therefore it woudl be useful to use an analogread. How can I detect how big my data is, that I want to measure ? is there a possibility to check the size of a string?

NO. All that happens internally.

Just use something like

void loop() {
data = analogRead(input);

so the uno collects data until the 512 byte is full and then writes it to the sd card ? did I understand that correctly ?

No. The SD library assembles the data you send to it using .write() or .print(), fills the buffer, and when the buffer is full, writes it to the card.

Best to study and run some of the library examples, and read the documentation, before starting this project.

ok, so from what I understand it works the following:

I open a file in setup and in the void loop I do the analogRead and print to sd but it is not every value on its own but it collects until a 512 byte block is full. then it is written to the sd card and so on until the measurement is finished right ?

The liminting time factor then is the time for analogRead ( about 100us) and the time X it takes to write the date correct ?

sorry for sounding stupid

Basically correct. The limiting time will be analog read (default 110 us) and however long it takes to write the occasional data block to the card (typically around 80 milliseconds, but up to 250 ms, I think).

For the outline I posted in reply#13, no data will be collected while the SD buffer "write to card" is happening.

This means you have to run the ADC in free running mode,
or there will be no time left for any SD activity.

And it is obvious that it is not possible to gather three values in the 100us.

So I will get a value every ~100us until buffer is full and then for another 80ms no data, since it is written to the SD card ? Or will there still be data captured in another buffer during those 80ms ?

What do you mean by free running mode?

Nope, this is a single processor system.