Trouble using char[] buffer for logging to SD card

Hello,

I am new to arduino, so please forgive my ignorance. If I am going about this in the entirely wrong fashion please let me know that also.

I am using an IR sensor to collect data. I need to collect this data at a frequency of roughly 700hz. The faster the better.

I am then storing this data in a csv file on an SD card. If I write the data straight to the file every time, I can write at about 90hz, which isn’t fast enough. I have been doing research and it looks like buffering the data to a char should be faster since I dont have to write to the file so often.

But when I do that my logging frequency stays about the same.

Are there any examples of how to achieve this? I’ve seen lots of forum posts discussing the matter, but no full examples.

Here is my code
-this code write 7 times to the char buffer, before sending that to the file. I would assume that it could log up to 7 times faster, but alas that is not the case.

#include <SPI.h>
#include <SD.h>

#define LCD_CS A3 // Chip Select goes to Analog 3
#define LCD_CD A2 // Command/Data goes to Analog 2
#define LCD_WR A1 // LCD Write goes to Analog 1
#define LCD_RD A0 // LCD Read goes to Analog 0

#define LCD_RESET A4

#define  BLACK   0x0000
#define BLUE    0x001F
#define RED     0xF800
#define GREEN   0x07E0
#define CYAN    0x07FF
#define MAGENTA 0xF81F
#define YELLOW  0xFFE0
#define WHITE   0xFFFF
#define N 128

/*
  Catalex - MicroSD Card Adaptor
  v1.0 11-01-2013
  The circuit:
  SD card attached to SPI bus as follows:

  MOSI - pin 51
  MISO - pin 50
  SCK  - pin 52
  CS   - pin 53
*/
const int CS = 53;

int sensor = 24;
const int notches = 48; // Total number of notches, black and white
const float wheeldia = 20; //In Inches, so change if different sized wheel
unsigned long StartTime = millis();
unsigned long CurrentTime;
unsigned long ElapsedTime;
unsigned long time;
int steps = 0;
float velocity;
float distance;
float wheelcirc;
float acceleration;
float velocity_old;
float ElapsedTime_old;
String FileName = String("empty");
File myFile;

char buffer[N];
int count = 0;


void setup()
{
  Serial.begin(19200);
  setupSDCard();

  Serial.println(F("Setup Done..."));
}






void loop()
{
  
  if (digitalRead(sensor) == HIGH)
  {
    writeToCharArray('1');
     Serial.println("writing to buffer1");
  }
  else
  {
    writeToCharArray('0');
    Serial.println("writing to buffer0");
  }

  if(count > 110)
  {
    //Write the char array to file
    for (int k = 0; k < count; k++) {
          myFile.print(buffer[k]);    
    }
    myFile.flush();
    Serial.println("flushed to file");
    
    //clear the char array
    for( int i = 0; i < count;  ++i ){
      buffer[i] = (char)0;
    }
    count = 0; //reset count
  }
}

void writeToCharArray(char value)
{
  char cstr[16];
  int j = 0;
  itoa(millis(), cstr, 10);
  for (j; j < 16; j++) //run through the array
  {
    if(cstr[j] != (char)0)//dont write empty places
    {
      buffer[count++] = cstr[j]; //copy the array into the buffer
    }
  }
  buffer[count++] = ',';
  buffer[count++] = value;
  buffer[count++] = '\r';
  
}

void setupSDCard()
{
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  Serial.print("Initializing SD card... ");
  if (!SD.begin(CS)) {
    Serial.println("Initialization failed");
    return;
  }
  Serial.println("Initialization done");

  getFileName(); //Find a free filename
  myFile = SD.open(FileName, O_CREAT | O_WRITE);//FILE_WRITE); // Open the file for writing
  if (myFile)
  {
      Serial.println(FileName + " opened successfully.");
  }
  else
  {
      Serial.println(FileName + " could not be opened.");
  }
}

void getFileName()
{
  for(int i = 1;i < 1000; i++)
  {
    Serial.println("checking for filename");
    if(!SD.exists((String)i + ".csv"))
    {
      FileName = (String)i + ".csv";
      break;
    }
  }
}

but no full examples.

Your code was not complete either. Please post your complete program

Sorry.

I've edited the original post

OP some questions and comments.

  1. Why are you writting a char at a time to the file, can you not simply write the char array to the file once? I do not use SD cards, but I can not imagine that they have you write on char at a time.
  2. You clear the array, this is a good thing, but there are faster ways of doing this. memset(buffer, 0, sizeof(buffer));
  3. writeTocharArray seems clunky, that for loop I see no need for it. You could reduce the logic like this.

//UNTESTED needs vetting.

memcpy(buffer + counter, cstr, strlen(cstr));
counter += strlen(cstr);

Hello, and thank you for your reply!!

The answer to most of your questions is that this is first program I have written and I am googling to try to find answers. I am trying to find the best way of doing things, but its difficult when I know so little.

  1. I will try to find a better way of writing to the file
  2. Thank you very much!
  3. Again thank you! I will try this out.

Best regards!

this code write 7 times to the char buffer, before sending that to the file.

In the code you posted you have

  if (count > 110)
  {
    //Write the char array to file
    for (int k = 0; k < count; k++)
    {
      myFile.print(buffer[k]);
    }

It looks to me as though this is writing rather more than 7 entries to the file

Sorry, let me be more clear there.

The code will write 7 log entries worth of text to the char before that char become full enough that it then writes to the file.

When it writes to the file, it writes whatever is in the char to the file.

When it writes to the file, it writes whatever is in the char to the file.

Why does it write 110 entries to the file when you have only put 7 entries in the array ?

Where is the 7 entries in the array limit checked in the code ?

Thank you for the great tips Romonaga! The solution was to write the entire buffer to the file rather than writing each character at a time to the file.

I can now get about 1400hz worth of logging with no delays inserted. If I put a delay(1) after each read then I get 550hz of logging.

The reason the frequency is not higher with the delay is because when it writes to the SD card I am spending some time writing instead of reading. I am missing about 12 ms worth of data every 32ms.

I am going to investigate interrupts and double buffering to solve this.

Thanks again!

BLakatos:
Thank you for the great tips Romonaga! The solution was to write the entire buffer to the file rather than writing each character at a time to the file.

Glad I could be of some help.