Writting an array to the SD card without looping through it

Hi!

I have to log some data, and I want to do it, as everybody, as fast as possible. Although I'm using a Mega, the ammount of RAM available is pretty limited. I use a SD card through the ethernet shield.

I'm looking for a fast efficient way of writting to a SD card. Right now, opening a file, writting something short and closing gives me around 350 writes in 5 seconds. If I leave the file opened, I can reach 2500 writes in the same 5 seconds, which is pretty good, but I bet I can do it better.

I've read posts about filling a buffer and writting a huge block at a time, reducing the SD accesses. The problem I find is that I really don't know how to write an entire array to the SD card. SD.write() seems to only work with strings or arrays of bytes, and using a for loop leaves me in the same position, with as many accesses to the sd card as positions has the array. It has to be a way to write an array of whatever data you like to the SD without having to loop through it, isn't it? I've been looking for a solution for this problem on and off for a couple of weeks, but I'm unable to find a solution.

With the SD library included with the IDE, this is about as fast as you will get.

byte testArray[128];

SD.write(testArray,128);

Thanks for answering so fast.

Yes, I noticed the write command, but I doesn't let me write anything that is not a String or an array of single bytes. I tried with an array of integer and it didn't compiled. The error was:

FastSDLogging.ino: In function 'void loop()':
FastSDLogging:70: error: no matching function for call to 'File::write(int [512], unsigned int)'
C:\Program Files (x86)\Arduino\libraries\SD/SD.h:35: note: candidates are: virtual size_t File::write(uint8_t)
C:\Program Files (x86)\Arduino\libraries\SD/SD.h:36: note:                 virtual size_t File::write(const uint8_t*, size_t)
C:\Program Files (x86)\Arduino\hardware\arduino\cores\arduino/Print.h:49: note:                 size_t Print::write(const char*)

Which I assume is telling me "You can do a write only with a uint8 array".

If you want help with that, you need to post your code.

Which I assume is telling me "You can do a write only with a uint8 array".

Or something that you have cast to a uint8_t array.

Are you trying to write the array as binary data or as ASCII data? That code (or the code with proper cast) will only write binary data. If you want ASCII data, you must iterate through the array.

The SD methods to write to a file already use buffering, so you are unlikely to see a huge improvement from implementing your own buffering techniques on top of the SD ones.

SurferTim, I didn't posted my code because it's pretty long and boring, and I don't want to make you lose your time looking at it. Basically, my idea is to read a sensor as fast as I can. I will try to take the elapsed time between captures with a timer.It works as expected, so my problem is not a coding issue, my problem is that I don't know how to do what I want to do :D.

So the thing is, I want to write fast some numeric data taken from a couple of sensors attached to the Mega board. Everywhere I looked, people recommended to buffer data and send a chunk of 512 bytes, but I really don't know how to do it, because they are int, not bytes or char. If I loop through the array with a for loop and use pritnln, I am just sending 2 bytes each time, isn't it? But from what PaulS says, if SD methods are buffered, it shouldn't be any difference between sending a chung of 512 bytes or sending 1 byte 512 times really quick, as long as you do not overflow the buffer.

Hell, this is driving me nuts. I think I will have to leave it as it is, and read every 5 ms or so and just write it, or just use as much RAM as I can and print it after the measuring cycle is finished. Is not that I need to achieve any specific speed, but I just wanted to know where the limits are.

Thanks for your help, anyway.

Well, what I finally did was set a timer to fire an interrupt every X us and in the ISR, a string with 3 comma-separated ints and raising a "write now" flag. The loop() was a "if writeFlag, then write". This proved faster than creating an array of 512x3 and writting all the array. I do not close the file after each write, which I suppose can be problematic if the card is extracted or the program doens't reach the close() statement.

With this approach, I was able to reliably write every 625 us, which I think is pretty fast. Of course, I suspect that depending on what you have to read (be it a ADC, a I2C sensor, etc) and the speed of your card will impact heavily on the results, but I'm happy with this.

I will post the code just in case someone finds it useful.

FastSDLogging.ino (3.4 KB)

Or even simpler, write directly to the file during the ISR.

Can someone confirm if there should be any issues with interrupt conflicts in this scenario?

Or even simpler, write directly to the file during the ISR.

Can someone confirm if there should be any issues with interrupt conflicts in this scenario?

Do not do this. Most of the time, you'll get away with it, because the write() and print() operations actually only involve writing to a buffer. But, when the buffer gets full, the data in the buffer needs to be transferred to the card, and that takes longer than should be done in an interrupt service routine.

There may (or may not) actually be interrupts involved in writing to the SD card. If so, then it would simply not be possible to write to the card during an ISR.

Do not do this.

Interrupts are not used in the SD libraries but there can be very large write latencies with SD cards. 50-100 milliseconds delays occasionally occur when the card is moving data for wear leveling, erasing flash, or other housekeeping.

Time in an ISR should be far less than a microsecond. SD.h takes longer than a microsecond every time the block buffer is written to the SD.

Interrupts are not used in the SD libraries

Isn't SPI dependent on interrupts?

Isn't SPI dependent on interrupts?

No, SdFat does not use interrupts to do SPI transfers. SD.h now uses SPI.h and is not interrupt based.

Full speed SPI is 8 MHz or one microsecond to transfer a byte. Using an ISR is not practical since the time to service an interrupt for each byte would be way too long.

As fat16lib said an sdcard has huge latencies from time to time (hard to say when) so it could happen the card does not take over your data for 10-250msecs quite often. You cannot avoid that.
What I am doing now with fat16lib's help is I've been logging data to my Sdcard as fast as I can with help of Nilrtos and NilFIFO.

I shoot 88bytes (an CSV line) up to 333x per second in the CSV format (500x works too with largest FIFO I can fit inside my 1284p but not 100% reliable yet). I've been using the fifo buffer 100 - 150 records deep.

#include <NilFIFO.h>
// Type for data record.
struct FifoItem_t {
	uint32_t tnow;   // CSV record (a single line) time in unix time
	uint16_t msnow;  // mseconds
	double ads;   // 16bit I2C sigma delta ADC
	double value[8];  // MCU ADC values filtered
	int16_t fifodatacnt;  // 
	int16_t fifofreecnt;  // 
	int16_t par1;  // param1
	int16_t par2;  // param2
};
NilStatsFIFO<FifoItem_t, 150> fifo;

When observing what is happening there I can see it requires up to ~100 fifo records from time to time in order to compensate SDcard's latency (normally up to 50). Been using the fast file.printField() for writing from the fifo to the sdcard.
The slower you write, the lower are the requirements on the fifo size. So you need to experiment.

,1378052487,275,0.049,81.45,87.72,90.27,91.24,91.53,90.85,90.64,83.74,-1,149,0,149,1812
,1378052487,278,0.049,81.45,87.72,90.27,91.24,91.53,90.85,90.64,83.74,-1,149,1,148,3400
,1378052487,281,0.049,80.10,86.13,88.62,89.62,89.90,89.28,89.13,82.55,0,148,0,149,1852

So the setup is:
1st task grabs data from adc etc., making filtering, smoothing etc. as fast as possible
2nd task (producer) is writing the current data to FIFO slots (in a regular sample period, ie. 4msecs = 250x per second)
3rd task (consumer) is writing the FIFO records to the SDcard ( with file.printField() )
4-6th task is starting/stopping the process (opening/closing the file on the sdcard when finished, so hot swap works fine).

Or even simpler, write directly to the file during the ISR.
Can someone confirm if there should be any issues with interrupt conflicts in this scenario?

Do not do this. Most of the time, you'll get away with it, because the write() and print() operations actually only involve writing to a buffer. But, when the buffer gets full, the data in the buffer needs to be transferred to the card, and that takes longer than should be done in an interrupt service routine.

Yes, I noticed it was not very reliable, thanks for the explanation. To tell you the truth, I was not really understanding what I was doing with the SD card. After your comments and with a bit of research, I think I see all the things I did wrong. Thanks for the Nil* suggestions. Although they look terrific, I think they might be overkill for my application, but might consider it in further projects.

I have a question about buffers. In the case of the Arduino writting to the SD card with the ethernet shield (if I'm not wrong, that's wiring the SD card directly to the arduino and use SPI), where is the buffer? is it a dedicated memory region on the microcontroller, is it a buffer I have to create and manage or is it a buffer in the SD card that the SD card manages? If the correct answer were the last, is there a way to know the size of this buffer? (some kind of SD diagnostic tool, I assume).

where is the buffer? is it a dedicated memory region on the microcontroller, is it a buffer I have to create and manage or is it a buffer in the SD card that the SD card manages? If the correct answer were the last

It is.

is there a way to know the size of this buffer?

You have the source code, so, of course, the answer is yes. Recognizing it might not be straightforward.

You can NOT make it smaller. The buffer MUST be the same size as the commit-to-file size, which is 512 bytes.

Hi,

Similar issue but, is there no way to write an array of 16 bit integers into the SDcard for datalogging?

#include <SPI.h>
#include <SD.h>
//assign pin
const int analogPin = 0;
//Sample rate in samples per second
 const float SAMPLE_RATE = 8000;  
//Sample Interval
 const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;
//SD chip select pin
const int chipSelect = 4;
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = -1;
//buffer definitions
#define BUFFSIZE 512
char buffer[BUFFSIZE];
char c;
int buffIndex=0, sensor=0;
//unsigned long start_times[105];
//unsigned long stop_times[105];

//-----------------------------------------------------------------------------------
 boolean logData() {
  boolean result = false;
  Serial.print("Initializing SD card...");
  pinMode(10, OUTPUT);

  // check if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    result = false;
  } 
  
  else {
    Serial.println("card initialized.");
    File dataFile = SD.open("f2.bin", FILE_WRITE);
    
    for(int i=0; i<1000; i++)
      {
     // start_times[i] = micros();
      sensor = analogRead(analogPin);    
      buffer[buffIndex++] = sensor; 
      if (buffIndex >= BUFFSIZE-1)
          {
           // buffer[buffIndex] = sensor; // terminate it with null at current index
            dataFile.write((uint16_t *) buffer, (buffIndex + 1)); //write the program buffer to SD lib backend buffer (PROBLEM IS HREE!!!!!!!!!!!!!)
            dataFile.flush();
            buffIndex = 0;     // reset buffer pointer
          }    
.....continued

I looked in the sd.h file, and noticed the write function does not support array of 16 bits integers.