SD card

Projects Programming
1

I’m working on a project where I would need to store data very faster. My goal would be to store 7 float variables, 1 long and 1 boolean (which would equal to 33 bits if I understand it right). I’m currently using an Arduino Pro Mini (ATMEGA328 328p 5V 16MHz), a “standard” microSD card adapter module (I got from Banggood) and a Samsung 32Gb micro SD card. The microSD card adapter is connected to the Arduino using SPI.

I use the following code to store the data on the SD card and to calculate how it take to store the data (using the standard Arduino SD library):

void SD_Card() {
  current_Time = micros();
  myFile = SD.open("test.txt", FILE_WRITE);
  if (myFile) {
    myFile.print(current_Time); myFile.print(",");
    myFile.print(raw_ax); myFile.print(",");
    myFile.print(raw_ay); myFile.print(",");
    myFile.print(raw_az); myFile.print(",");
    myFile.print(raw_gx); myFile.print(",");
    myFile.print(raw_gy); myFile.print(",");
    myFile.print(raw_gz); myFile.print(",");
    myFile.print(altitude);myFile.print(",");
    myFile.println(max_Altitude);
    myFile.close(); // close the file
  }
  // if the file didn't open, print an error:
  else {
    Serial.println("error opening test.txt");
  }

  long interval = micros() - current_Time;
  Serial.println(interval);
}

Here you can see how long it takes to store the data in microseconds (about 17’000 to 20000 microseconds):

17852
20328
16380
17760
20456
20404
20444
16300
17704
20288
16420
17684
20380
16312
17748
20256
16224

I read that using a different library, in this case the “SdFat-master”, could give better result, which it did as it took less time to store the data (about 8’200 to 15’000 microseconds):

12336
8200
9640
12496
8156
9724
12376
8356
9748
12192
8248
15444
8300
9796
12256
8300

I would need to store the date in maximum 5000 microseconds (5 milliseconds), ideally even faster, is there a way to speed it up ? Is there something I could change in the setup section to increase the speed ? Is there a way to store the data differently, for example by merging the data in one single string, so I would need to print only one time instead of 9 times?

An other option would be to store the data in a different way (using eeeprom, flash, or anything else) and then to transfer the data to the SD card ? If so what type of memory should I use ?
Is the Arduino the issue ? Could I store the data faster I would use a faster micro-controller like the Teensy 3.5 ?

Any help appreciated. Thanks.

2

What volume of data do you need to store and how often is it updated ?

3

I need to store 9 variables, 7 floats, 1 long and 1 boolean so about 33 bytes of data every 10 millisecond (100 times a second) for about 10 -15 seconds.

1 Like
4

Try writing the data for the requisite number of cycles without closing the file for each line.

Might help to write out the binary data only, drop the commas.

5

Thanks for your help, your input was useful.
I tried without opening and closing the file each time and it working much better, I’m know getting about 2100 microseconds per loop, which is below my targeted 5000 microseconds.

I didn’t drop the commas since I need them for later use in Excel.

What do you mean by “ write out the binary data only” ? Should I convert my variables into binary once i calculated them?

Is there still a way to make it even faster ? Like combining all data into a single string ?

6

I think the 328P is unlikely to be the problem here. At 16 MHz, it's not slowing you down.

You might see if the SD or SPI libraries allow you to adjust the SPI transfer speed. Maybe speeding that up would be help.

But you know, you really aren't transferring very much data at a time, and I suspect there is just a lot of overhead and delay dealing with commands sent to the SD card, and the requirements of FAT32. If you're saving all this to a file, then you have to deal with the file system, which includes navigating to clusters, etc. Then there is the issue of what it takes to write data to the SD card - if you add a few bytes to the end of a file, does it have to erase that whole sector first?

But a few suggestions:

First, saving 33 bytes at a time is just going to complicate things. You want a complete entry to be a fixed number of bytes, and the entry should be a size that divides evenly into 512, so that no entry ever has to cross a sector boundary. Have you looked at the file you are producing in a hex editor?

Second, I would certainly try using a single myFile.print that includes the entire entry with all the pieces concatenated into one string.

Third, I don't know how you would do it in C++, but in theory you could accumulate 512 bytes of entries in a block of ram, then write that block to the SD card as a full sector.

7

I was thinking that if speed is really important, you could write out the binary representation of your variables i.e. four bytes for a float. As it is you're printing text, including a decimal point and a comma to separate. It would be marginally faster too, if you put all the variables in a struct and write it out in one call.

However, then you would need something to parse it into CSV form for Excel to read.

8

ShermanP:
Third, I don't know how you would do it in C++, but in theory you could accumulate 512 bytes of entries in a block of ram, then write that block to the SD card as a full sector.

The SD library does 512 byte buffering for you out of the box.

9

If you don't close/open the file for every line, you get a higher speed on average, but there will be bigger differences, and once every 512 characters writing will take those 12 ms you noticed with SdFat library anyway.

That's because SD card writing is organized in blocks of 512 bytes and the libraries handle that for you already, if you don't call close() or flush().

So all depends on your real requirement: If you need a strict 10 ms cycle, SD via SPI is not your solution.

BTW: print does not write binary, but text. As such, your 9 variables will be much more than 33 bytes.

10

I never concatenated before but I tried doing this:

char text[60];
sprintf(text, "%d,%d,%d,%d,%d,%d,%d,%d,%d\n", increment, raw_ax, raw_ay, raw_az, raw_gx, raw_gy, raw_gz, altitude, max_Altitude);

and than write only one time. It is faster this way as you guys recommended. Here are the results in microseconds (about 100 microseconds):

100
100
100
100
100
100
100
100
100
100
100
104
1492
100
104
100
100
108
96
100
104
100
100
100
104
1552
100
104
104
108
100
100
100
104
104
100
104
104
1492

I notice an increased time every 12 print, is it what michael_x is saying ?

I still have an issue, how do I use "sprintf" for other variables than integer (for example boolan and floats) ?

Because if I print "text", I see that the last two numbers are wrong, one should be an boolean and the last a float.

0,-36,-80,16396,14,-9,-20,5791,17692

0,12,48,16396,-4,-18,36,5791,17692

0,-196,-12,16312,4,-8,19,5791,17692

0,36,72,16316,9,0,39,5791,17692

0,-28,68,16424,-8,16,70,5791,17692

How can I change this ?

I want to thank all of you guys one more time since thanks to your help I was able to get from about 8'200 - 15'000 microseconds to about 100 !

11

I still have an issue, how do I use "sprintf" for other variables than integer (for example boolan and floats) ?

A boolean is an integer so use %d for it in the format string. Floats are not enabled in the Arduino C++, but you can convert to string with the dtostrf() function and then use %s (string) in the format string.

int x = 20;
int y = 30;
boolean z = true;
float num = 3345.14159;
char buffer[32];  

void setup()
{
   Serial.begin(115200);
   char numstr[10];
   dtostrf(num, 2,5,numstr);
   sprintf(buffer, "%d,%d,%d,%s", x,y,z,numstr);
   Serial.println(buffer);
}

void loop()
{
   
}
12

Thanks groundFungus, it works perfectly now

13

RedIron:
I need to store 9 variables, 7 floats, 1 long and 1 boolean so about 33 bytes of data every 10 millisecond (100 times a second) for about 10 -15 seconds.

33 bytes x 100/s x 15 sec = 49,500 bytes total? I do the math right?

A teensy has 64,000 bytes of RAM. Just use a teensy, write it all to an array then when your done. Send it off to your SD drive.

-jim lee

14

100Hz is tough when the SD card sometimes takes 12ms to complete an operation. You must close the file or sync() the file system regularly, otherwise the file directory never gets updated and the file appears to be empty when you look at it on a computer with a regular file system. That synchronization is usually the most time-consuming thing that the SD card does.

SdFat does allow you to do work while it's waiting for the SD card. You can even use the same SPI bus that the card is using. The trick is the yield() function.

Be careful with yield(). The Arduino actually calls yield() BEFORE setup(), so you need to set a global variable at the end of setup() and don't try to do any processing inside yield() until that is set.

15

Would it be possible to create the file ahead of time, and fill it with all FFs, and then just "overwrite" the file from the beginning with data? You wouldn't ever have to close the file.

16

That works for this application but usually you're logging with no defined end time. You might gather gigabytes of data over weeks of logging.

17

I'm closing the file after 2000 cycles (which equals to 20 seconds). I looked at the data on the SD card and I do not see any "mistakes", every variable has been stored correctly.

Regarding time, storing the data takes about 100 microseconds with "pics" of about 1500 microseconds every 12 cycles. I do not see those 12ms you are talking about any more.

Regarding using a Teensy, I think it's a good idea if as you calculated I do not exceed 15 sec. I expect about 10 to 15 seconds of data gathering but it might be more at the end so I will stick with my current method.

ShermanP:
Would it be possible to create the file ahead of time, and fill it with all FFs, and then just "overwrite" the file from the beginning with data? You wouldn't ever have to close the file.

I like the idea how could this be implemented ?

I still have the question regarding other methods for storing fast data, is there another way / form of data storage that would give the same or even better results regarding logging time ?

18

How about Ethernet? Offload the responsibility for storing the data to some other computer. A Pi, or your PC.

19

RedIron:
I'm closing the file after 2000 cycles (which equals to 20 seconds). I looked at the data on the SD card and I do not see any "mistakes", every variable has been stored correctly.

I'm not sure it makes any difference, but as I said before, it might save some time if all the entries used the same number of bytes, and if that number divided evenly into 512 so no entry would span a sector boundary. I just don't know how the SD library does all this stuff, so I don't know if that would save any time.

Would it be possible to create the file ahead of time, and fill it with all FFs, and then just "overwrite" the file from the beginning with data? You wouldn't ever have to close the file.

I like the idea how could this be implemented ?

I think you would need special software to prepare the card. You would have to save a number of files to the card, each up to 4GB in size assuming you're using FAT32, but then ideally the entire data area of the card would need to be erased so the card controller would know it didn't need to erase again when you write data. And then I don't know how the SD library handles overwriting at a pointer. You might have to write your own code. So this could be pretty complicated unless someone has already done it and there's a library.

I still have the question regarding other methods for storing fast data, is there another way / form of data storage that would give the same or even better results regarding logging time ?

You could have an intermediate device that would be responsible for saving the data, but which could continue to accumulate incoming data while the writing is going on. You could connect to that device via serial, ethernet, or even wireless. But if you are able to stop collecting data at some point and take whatever time is needed to write it out to the SD, then an intermediate serial ram chip might work. But I think they max out at 125KB, which it looks like wouldn't be nearly big enough.

Edit: Reading jimLee's comment again, I see I was wrong about a serial ram not being large enough. It looks like 125KB would give you up to 30 seconds of data before you had to stop and save everything.

20

One more comment about the general problem. If the data generation process could be interrupt driven, then it might be possible to collect the data at a precise regular rate and save it to one of two ram buffers of 512 bytes each. The ISR would keep a pointer to where it is in the buffer, and when the current one is filled up, it would set a bufferFullx flag and switch to saving to the other buffer.

Then the Loop would do nothing but check the bufferFullx flags, and when it finds one set, it would print the contents to the SD card in the CSV text format, then clear the flag. The SD file would be closed only after the full 15 seconds, or whatever, has elapsed.

This would work if it never takes longer to do the SD write than it takes to fill up a buffer with raw data, and if the time required by the ISR is never long enough to mess up any SD write in progress. I think SPI as a synchronous format should actually be pretty tolerant of these minor interrupt delays.

If raw data (the 33 bytes) is saved in the buffers, the buffers could be smaller than 512 bytes. And really there's nothing magic about 512 here. At 33 bytes each, that would give you about 15 readings, but the real issue is how many readings is equal to the maximum SD write time. You would want the buffer size to be at least that big, but not much bigger given the ram constraints of the 328P (SD or SDFat will also want some ram). But you could always fall back to a Teensy if available ram is an issue.

I don't know how you would dedicate ram to these buffers, or address them, or read data from them for conversion to CSV, in C++, but I assume there is a way to do all that if you are a C++ jock. And actually, I would love to see an example of that code if anybody knows of one.