Improve Datalogging frequency

ZNaught · June 20, 2015, 10:29pm

I am working on this project to control solenoids and datalog. For the solenoid control, I need to be able to pull the data and filter it as quickly as possible. This all needs to be written to an SD card for analysis later. This is for a rocket and thus I cannot watch serial data or determine if the software is working properly until post-flight analysis (after I am happy with the software of course). All sensor data comes from the Adafruit 10DOF sensor and I am using the Sainsmart microSD adapter.

I am new to Arduino programming, however, I have had experience with Java and other higher level languages (a bit rusty though).

I originally implemented the standard SD libraries, however, I was only getting about 15 Hz from this. I then switched over to a similar implementation using SdFat and now it is about 20 Hz. Just using the barebones implementation (just writing the time, no actual data) I only reached 70 Hz. Testing the sdcard using the SDFat "LowLatencyLogger" example, I was able to achieve 500 Hz, but this is without customizing it for my own use. I have absolutely no idea where to start in implementing that method for the 10DOF data and even then I suspect the number of cycles going into retrieving that data will limit that heavily.

Here is what I have so far:

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_LSM303_U.h>
#include <Adafruit_BMP085_U.h>
#include <Adafruit_L3GD20_U.h>
#include <Adafruit_10DOF.h>
#include <SPI.h>
#include <SdFat.h>
#include <SdFatUtil.h>

/*** CONSTANTS ***/

/* Assign a unique ID to the sensors */
Adafruit_10DOF                dof   = Adafruit_10DOF();
Adafruit_LSM303_Accel_Unified accel = Adafruit_LSM303_Accel_Unified(30301);
Adafruit_LSM303_Mag_Unified   mag   = Adafruit_LSM303_Mag_Unified(30302);
Adafruit_BMP085_Unified       bmp   = Adafruit_BMP085_Unified(18001);

const int chipSelect = 4; // digital pin for SD Card writing

/* Update this with the correct SLP for accurate altitude measurements */
float seaLevelPressure = SENSORS_PRESSURE_SEALEVELHPA; // sea level pressure in hPa

const int dataTypes = 7 * 2; // Number of data types to log, update if number of outputs changes, multiplied by 2 for commas for csv


/*** INITIALIZE VARIABLES ***/
String dataLine[dataTypes]; // Stores line of data for datalogger
SdFat SD;
SdFile File;

/*** FUNCTIONS ***/

/* Writes to datalog file */
void dataLogWrite(String data[])
{
  String dataString = "";

  for (int i = 0; i < dataTypes; i++)
  {
    dataString += data[i];
  }

  // open the file for write at end like the Native SD library
  if (!File.open("test.csv", O_RDWR | O_CREAT | O_AT_END)) {
    SD.errorHalt(F("opening test.csv for write failed"));
  }
  // if the file opened okay, write to it:
  File.println(dataString);

  // close the file:
  File.close();
}


void setup(void) 
{
  Serial.begin(9600);

  /* see if the card is present and can be initialized: */
  if (!SD.begin(chipSelect, SPI_FULL_SPEED)) {
    Serial.println(F("Card failed, or not present"));
    SD.initErrorHalt();
  }

  
  /* Initialise the sensor */
  if(!accel.begin())
  {
    /* There was a problem detecting the ADXL345 ... check your connections */
    Serial.println(F("No LSM303 detected"));
    while(1);
  }
  if(!bmp.begin())
  {
    /* There was a problem detecting the BMP085 ... check your connections */
    Serial.print(F("No BMP085 detected"));
    while(1);
  }
  
  // Write column titles
  dataLine[0] = F("Time(s)");
  dataLine[1] = F(",");
  dataLine[2] = F("x(m/s^2)");
  dataLine[3] = F(",");
  dataLine[4] = F("y(m/s^2)");
  dataLine[5] = F(",");
  dataLine[6] = F("z(m/s^2)");
  dataLine[7] = F(",");
  dataLine[8] = F("Temp(C)");
  dataLine[9] = F(",");
  dataLine[10] = F("Press(hPa)");
  dataLine[11] = F(",");
  dataLine[12] = F("Alt(m)");
  dataLine[13] = F(",");

  dataLogWrite(dataLine);

}

void loop(void) 
{
  
  /* Get a new sensor event */ 
  sensors_event_t eventBaro; 
  sensors_event_t eventAccel; 
  accel.getEvent(&eventAccel);
  bmp.getEvent(&eventBaro);
  float temperature;
  bmp.getTemperature(&temperature);
  
  // Create new line of data
  dataLine[0] = String(millis()/1000.0,4);
  dataLine[1] = F(",");
  dataLine[2] = String(eventAccel.acceleration.x,4);
  dataLine[3] = F(",");
  dataLine[4] = String(eventAccel.acceleration.y,4);
  dataLine[5] = F(",");
  dataLine[6] = String(eventAccel.acceleration.z,4);
  dataLine[7] = F(",");
  dataLine[8] = String(temperature,4);
  dataLine[9] = F(",");
  dataLine[10] = String(eventBaro.pressure,4);
  dataLine[11] = F(",");
  dataLine[12] = String(bmp.pressureToAltitude(seaLevelPressure,eventBaro.pressure),4);
  dataLine[13] = F(",");

  dataLogWrite(dataLine);
  
}

Any suggestions on how to drastically improve the datalogging performance? I still need to implement the actual control logic which is the main purpose of the project.

Thanks!

system · June 20, 2015, 10:32pm

Any suggestions on how to drastically improve the datalogging performance?

Quit pissing away time and resources using Strings.

ZNaught · June 20, 2015, 10:34pm

PaulS:
Quit pissing away time and resources using Strings.

I have read you suggesting this before, I did my best reducing it previously. How do I avoid using it for writing to .csv?

system · June 20, 2015, 11:14pm

How do I avoid using it for writing to .csv?

You are kidding yourself if you think you are writing to a .csv file. You are NOT. You are writing to a buffer. When the buffer gets full, the SD class transfers the data in the buffer to the file.

So, write the data ONE VALUE AT A TIME, to the buffer. Do not waste time, effort, or memory, putting together an array of characters or, even worse, a String. You will transfer data to the buffer NO FASTER.

An array of Strings does NOT do what the comment suggests.

ZNaught · June 21, 2015, 7:20am

PaulS:
You are kidding yourself if you think you are writing to a .csv file. You are NOT. You are writing to a buffer. When the buffer gets full, the SD class transfers the data in the buffer to the file.

So, write the data ONE VALUE AT A TIME, to the buffer. Do not waste time, effort, or memory, putting together an array of characters or, even worse, a String. You will transfer data to the buffer NO FASTER.

An array of Strings does NOT do what the comment suggests.

Thanks for the info, I think I understand now. I am still closing the file after each sensor event, does this matter or not?

Here is my new code:

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_LSM303_U.h>
#include <Adafruit_BMP085_U.h>
#include <Adafruit_L3GD20_U.h>
#include <Adafruit_10DOF.h>
#include <SPI.h>
#include <SdFat.h>

/*** CONSTANTS ***/

/* Assign a unique ID to the sensors */
Adafruit_10DOF                dof   = Adafruit_10DOF();
Adafruit_LSM303_Accel_Unified accel = Adafruit_LSM303_Accel_Unified(30301);
Adafruit_LSM303_Mag_Unified   mag   = Adafruit_LSM303_Mag_Unified(30302);
Adafruit_BMP085_Unified       bmp   = Adafruit_BMP085_Unified(18001);

const int chipSelect = 4; // digital pin for SD Card writing

/* Update this with the correct SLP for accurate altitude measurements */
float seaLevelPressure = SENSORS_PRESSURE_SEALEVELHPA; // sea level pressure in hPa

/*** INITIALIZE VARIABLES ***/
SdFat SD;
SdFile File;

/*** FUNCTIONS ***/

/* Writes to datalog file */
void titleWrite()
{
  // open the file for write at end like the Native SD library
  if (!File.open("test.csv", O_RDWR | O_CREAT | O_AT_END)) {
    SD.errorHalt(F("opening test.csv for write failed"));
  }
  // if the file opened okay, write to it:
  File.print("Time(s)");
  File.print(",");
  File.print("x(m/s^2)");
  File.print(",");
  File.print("y(m/s^2)");
  File.print(",");
  File.print("z(m/s^2)");
  File.print(",");
  File.print("Temp(C)");
  File.print(",");
  File.print("Press(hPa)");
  File.print(",");
  File.print("Alt(m)");
  File.print(",");
  File.println("");

  // close the file:
  File.close();
}

void dataWrite()
{
  /* Get a new sensor event */ 

  sensors_event_t eventBaro; 
  sensors_event_t eventAccel; 
  accel.getEvent(&eventAccel);
  bmp.getEvent(&eventBaro);
  float temperature;
  bmp.getTemperature(&temperature);


  // open the file for write at end like the Native SD library
  if (!File.open("test.csv", O_RDWR | O_CREAT | O_AT_END)) {
    SD.errorHalt(F("opening test.csv for write failed"));
  }
  // if the file opened okay, write to it:
  // Create new line of data
  File.print(millis()/1000.0,4);
  File.print(",");
  File.print(eventAccel.acceleration.x,3);
  File.print(",");
  File.print(eventAccel.acceleration.y,3);
  File.print(",");
  File.print(eventAccel.acceleration.z,3);
  File.print(",");
  File.print(temperature,2);
  File.print(",");
  File.print(eventBaro.pressure,3);
  File.print(",");
  File.print(bmp.pressureToAltitude(seaLevelPressure,eventBaro.pressure,3));
  File.print(",");
  File.println("");

  // close the file:
  File.close();
}


void setup(void) 
{
  Serial.begin(9600);

  /* see if the card is present and can be initialized: */
  if (!SD.begin(chipSelect, SPI_FULL_SPEED)) {
    Serial.println(F("Card initialize failure"));
    SD.initErrorHalt();
  }

  
  /* Initialise the sensor */
  if(!accel.begin())
  {
    /* There was a problem detecting the ADXL345 ... check your connections */
    Serial.println(F("No LSM303 detected"));
    while(1);
  }
  if(!bmp.begin())
  {
    /* There was a problem detecting the BMP085 ... check your connections */
    Serial.print(F("No BMP085 detected"));
    while(1);
  }
  
  // Write column titles
  titleWrite();

}

void loop(void) 
{
  

  dataWrite();
  
}

I am not getting around 18-20 Hz per set of data. Anyway to increase this further?

Thanks!

system · June 21, 2015, 12:45pm

I am still closing the file after each sensor event, does this matter or not?

Yes. Each time you close the file, you must flush the buffer to the file. Can you guess which part of writing to the file is the slowest? It is not writing to the buffer...

  File.print(millis()/1000.0,4);

Doing the calculation on the other end would be orders of magnitude faster.

void loop(void) 
{
  dataWrite();
}

This function call looks pretty useless to me. If you are going to put everything in one function, it makes sense that that function be called loop().

  accel.getEvent(&eventAccel);
  bmp.getEvent(&eventBaro);
  float temperature;
  bmp.getTemperature(&temperature);

Are any of these functions blocking? That is, how long does it take to execute these three function if you do not open, write to, and close the file? If it is these functions that take a majority of the time, then optimizing writing to the file makes little sense.

ZNaught · June 21, 2015, 5:15pm

PaulS:
Yes. Each time you close the file, you must flush the buffer to the file. Can you guess which part of writing to the file is the slowest? It is not writing to the buffer...
  File.print(millis()/1000.0,4);
Doing the calculation on the other end would be orders of magnitude faster.
void loop(void) 

{
dataWrite();
}



This function call looks pretty useless to me. If you are going to put everything in one function, it makes sense that that function be called loop().

accel.getEvent(&eventAccel);
bmp.getEvent(&eventBaro);
float temperature;
bmp.getTemperature(&temperature);



Are any of these functions blocking? That is, how long does it take to execute these three function if you do not open, write to, and close the file? If it is these functions that take a majority of the time, then optimizing writing to the file makes little sense.

Yeah, definitely seems like the root of the problem is the Adafruit 10DOF libraries. Grabbing an event from them is about 0.041 s, so the writing is definitely not the bottleneck. I'll see if I can go about optimizing those files.

Thanks for the help!

EDIT: I removed File.close entirely from my code and it simply doesn't write at all to the file. How frequently should I close the file and reopen it?

system · June 22, 2015, 1:00pm

EDIT: I removed File.close entirely from my code and it simply doesn't write at all to the file. How frequently should I close the file and reopen it?

How much data can you afford to lose? If the answer is none, then open and close the file every time.

If the answer is "one second's worth", then, every second, close the open file, and open it again.

Topic		Replies	Views
Can't add blank line to a SD file Programming	22	153	April 18, 2026
Performance Optimization (Number of Loops per Second) Programming	11	157	January 20, 2025
Log data to the SD card/SD card speed Storage	10	5753	July 16, 2022
Arduino SD Card write issue - intermittent logging/creating file Programming	28	731	July 7, 2024
SdFat.h Datalogging Speed Boost? Storage	24	3942	May 6, 2021

Improve Datalogging frequency

Related topics