Go Down

Topic: logging data to an SD card efficiently (Read 1 time) previous topic - next topic

inkosi

Mar 12, 2017, 01:56 pm Last Edit: Mar 12, 2017, 02:07 pm by inkosi
Hi,

I am reading data from an accelerometer (all X, Y and Z) values at a rate of 100 Hz.

I need to log this data to an SD card. So my initial code to do that was simple. Just grab the acceleration data and keep it in a string with the data delimited by a comma:

Code: [Select]
#include <Wire.h>
#include <SPI.h>
#include <Adafruit_LSM9DS0.h>
#include <Adafruit_Sensor.h>
#include <SD.h>
String dataString;
unsigned long timestamp;

File logfile;
Adafruit_LSM9DS0 lsm = Adafruit_LSM9DS0();

void setupSensor()
{
  lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_2G);
}

void setup()
{
  Serial.begin(9600);
  Serial.println("LSM raw read demo");
 
  // Try to initialise and warn if we couldn't detect the chip
  if (!lsm.begin())
  {
    Serial.println("Oops ... unable to initialize the LSM9DS0. Check your wiring!");
    while (1);
  }
  Serial.println("Found LSM9DS0 9DOF");
  Serial.println("");
  Serial.println("");

  if (!SD.begin(4))
  {
    Serial.println("No SD card detected. Data will be transmitted, but not stored."); // No card available? Let us know.
  }
  else
  logfile = SD.open("Data.txt", FILE_WRITE);
}

void loop()
{

    sensors_event_t accel;
    lsm.readAccel();
    uint32_t timestamp = micros();
    lsm.getAccelEvent(&accel,timestamp); 

     dataString =
     timestamp + String(",")
     + accel.acceleration.x + String(",")
     + accel.acceleration.y + String(",")
     + accel.acceleration.z + String(",");

     logfile.println(dataString);
     logfile.flush();
}


So I have a wireless SD shield, put the SD card in and run the code. However writing to the SD card is extremely slow this way. I took an average of all intervals and that came out to be approximately 40 Hz. I have included this file as an attachment as LS.txt to show you what it looks like. But my rate is being halved or more. I figured this is because storing all the data in a string is extremely slow and I read some threads here and thread and decided to use a struct, like so:

Code: [Select]
#include <Wire.h>
#include <SPI.h>
#include <Adafruit_LSM9DS0.h>
#include <Adafruit_Sensor.h>
#include <SD.h>
String dataString;
unsigned long timestamp;

File logfile;

struct datastore {
    unsigned long ts;
   int16_t ax;
   int16_t ay;
   int16_t az;
};


struct datastore  myData;
byte *buff = (byte*)&myData;

Adafruit_LSM9DS0 lsm = Adafruit_LSM9DS0();

void setupSensor()
{
  lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_2G);
}

void setup()
{
  Serial.begin(9600);
  Serial.println("LSM raw read demo");
 
  // Try to initialise and warn if we couldn't detect the chip
  if (!lsm.begin())
  {
    Serial.println("Oops ... unable to initialize the LSM9DS0. Check your wiring!");
    while (1);
  }
  Serial.println("Found LSM9DS0 9DOF");
  Serial.println("");
  Serial.println("");
  if (!SD.begin(4))
  {
    Serial.println("No SD card detected. Data will be transmitted, but not stored."); // No card available? Let us know.
  }
  else
  logfile = SD.open("clc.txt", FILE_WRITE);
}

void loop()
{

    sensors_event_t accel;
    lsm.readAccel();
    uint32_t timestamp = micros();
    lsm.getAccelEvent(&accel,timestamp);
   
    for (int i = 0; i < 500; i++)
    {
        myData.ts = micros();
        myData.ax = accel.acceleration.x;
        myData.ay = accel.acceleration.y;
        myData.az = accel.acceleration.z;
       
      logfile.write((byte*)&myData, sizeof(datastore));
    }
   logfile.close();
}


The problem is, while it appears to be working (file is saved on the SD card and there is some data inside it), the notepad file is saved with what I assume is garbled data. I've attached this file as clc.txt. I think this is because it just saves byte data and not a string and notepad can't read it. I want this to be readable just like the file with the saved string data (first example).

So my questions are: 1) is this approach of using a struct correct to efficiently save the data? and 2) if so, how can I get readable data on my file. if not, how should I go about saving these "packets" of data every 100 Hz?

Regards,

inkosi

I have discovered that, if I replace the code that saves the string data by only calling flush() after a certain number of calls like so:

Code: [Select]
    sensors_event_t accel;
    lsm.readAccel();
    uint32_t timestamp = micros();
    lsm.getAccelEvent(&accel,timestamp); 

    for (int i = 0; i < 500; i++)
    {

     dataString =
     micros() + String(",")
     + accel.acceleration.x + String(",")
     + accel.acceleration.y + String(",")
     + accel.acceleration.z + String(",");
          logfile.println(dataString);
    }
    logfile.flush();


It is a lot faster (~800 Hz). But I got 33000 packets of data as opposed to 500, why is that? And also,  how is an 800 Hz rate possible when the data coming from the acceleromter is 100 Hz and what exactly does it mean?

jremington

If you write binary, as in the second example, you have to read binary. Notepad can't do that.

Avoid Strings, they cause memory problems.

I would sprintf() to format the data into a character array, and write the corresponding character array, e.g.

Code: [Select]
char buffer[20];
sprintf(buffer,"%d,%d,%d", ax,ay,az);  //assuming ax, etc are ints
logfile.println(buffer);

cattledog

I agree with jremington that you should keep your data at character strings, rather than binary. You should be able to match the 100Hz acquisition rate.

Quote
However writing to the SD card is extremely slow this way. I took an average of all intervals and that came out to be approximately 40 Hz.
The SD library uses a buffer to physically write to the card in large blocks. logfile.print() just places the data in a buffer. There is no need to use file.close() or file.flush(), commands which physically write to the card and empty the buffer, after every placement of data into the buffer.

To avoid excess data records, only call a logfile.print() when you have new data to record and place in the buffer.

Call file.close() when you are done. Let the library and the card controller control the timing of the physical storage. 

athome

Hi, great information. Thank you!
I wrote a logger Software too and used 'til now file.flush() during the procedure in this way
Code: [Select]

LogFile.print(status);
  LogFile.print(";");
  LogFile.print("0");
  LogFile.print("\r\n");
  LogFile.flush();


The value rate is only around 10Hz. So I'm not so time critical.
But after a short while, not more than 100 write cycles (with only a few KB), my SanDisk 8GB micro SD card got defect sectors! I can reformat the card on a Windows machine and afterwards I don't see any defect sectors.
But after a few write cycles on Arduino the data are not written anymore and the defect sectors are back!
So the question is: is it just a defective card? Or is the problem in doing a file.flush()
I cannot do a file.close() because I have no defined stop! I remove the battery and then the file has to be closed! Is there an error in doing it so?

bnschmn

I have discovered that, if I replace the code that saves the string data by only calling flush() after a certain number of calls like so:

Code: [Select]
    sensors_event_t accel;
    lsm.readAccel();
    uint32_t timestamp = micros();
    lsm.getAccelEvent(&accel,timestamp); 

    for (int i = 0; i < 500; i++)
    {

     dataString =
     micros() + String(",")
     + accel.acceleration.x + String(",")
     + accel.acceleration.y + String(",")
     + accel.acceleration.z + String(",");
          logfile.println(dataString);
    }
    logfile.flush();


It is a lot faster (~800 Hz). But I got 33000 packets of data as opposed to 500, why is that? And also,  how is an 800 Hz rate possible when the data coming from the acceleromter is 100 Hz and what exactly does it mean?
Hey inkosi,

Any update on the final operative sketch? Would be grand if you share it

Go Up