Data Logging to SD card using an interrupt at 5-10 kHz

Using Arduino Storage
1

Hello everyone!

I need to log 1 byte at 5kHz speed to SD card, it's like 9 bit input, one of them is clock and I use it for interrupt. Here is the code:

#include <SD.h>
#include <SPI.h>  

File dataFile;

void setup() {
  Serial.begin(250000);
  Serial.print("Initializing SD card...");
  if (!SD.begin(10, 11, 12, 13)) {
    Serial.println("Card failed, or not present");
    return;
  }
  Serial.println("card initialized.");
  
  attachInterrupt(2, beep, RISING);
  if (SD.exists("datalog.txt")) {SD.remove("datalog.txt");}
  dataFile = SD.open("log.hex", FILE_WRITE);
  DDRF = B00000000;
}

void loop() {  }

void beep() {  dataFile.write(PINF); }

there is another arduino which generates bytes and I can control the frequency. Here is the code for that too:

int var =0;
bool output = HIGH;

void setup() {
DDRD = DDRD | B10000000;
}

void loop() {
if (output == HIGH) {
    PORTD = byte(var);
   var=var + 1;
  }
  PORTD = PORTD ^ B10000000;
  output = !output;
  delayMicroseconds(1000);
}

I know it is not perfect but at least it implements the device that sends bytes.

The default SD library is too slow, as I need to log each byte for less that 200 microseconds.

I use Arduino Mega with Adafruit data logging shield. So the main question is, how can I improve the logging speed?

I read the fat16lib's topic about logging 40 000 samples per second, but he uses timer for logging and I need to use an interrupt. I could not understand the example he wrote, as you can see, noobie here. A little guidance on what to learn or what to change would help, as it is not the first time i get into algorithms.

Thanks in advance!

Regards,
Javid Yagubali

2 
  if (!SD.begin(10, 11, 12, 13)) {

Which version of the SD library takes 4 arguments for the begin() method?

  if (SD.exists("datalog.txt")) {SD.remove("datalog.txt");}
  dataFile = SD.open("log.hex", FILE_WRITE);

If one file exists, delete it. Then, open or create a different file. Hmmm...

The default SD library is too slow, as I need to log each byte for less that 200 microseconds.

Most of the time, the value will simply be written to a buffer. That won't take anywhere near 200 microseconds.

When the buffer gets full, it needs to be committed to the file, and THAT takes time. Nothing you can do about it on a low-memory, single processor system.

So the main question is, how can I improve the logging speed?

No, the main question is where is the code you are actually running? The secondary question is how many samples per second you are actually getting? The tertiary question is how do you know that?

3

javid-y:
Hello everyone!

I need to log 1 byte at 5kHz speed to SD card, it's like 9 bit input, one of them is clock and I use it for interrupt.

I know it is not perfect but at least it implements the device that sends bytes.

The default SD library is too slow, as I need to log each byte for less that 200 microseconds.

I use Arduino Mega with Adafruit data logging shield. So the main question is, how can I improve the logging speed?

I read the fat16lib's topic about logging 40 000 samples per second, but he uses timer for logging and I need to use an interrupt. I could not understand the example he wrote, as you can see, noobie here. A little guidance on what to learn or what to change would help, as it is not the first time i get into algorithms.

Thanks in advance!

Regards,
Javid Yagubali

use the RAM of the MEGA as a few buffers, Roll through the buffers in the ISR, as each buffer is exhausted, mark it as dirty. In the main loop write the dirty buffers using file.write(), clear the dirty flag after the buffer has been written to the SD card.

#define BUFFSIZE 512
#define BUFFCNT 4
#define stopPin 4 //someArduinoPinToStopLogging

typedef struct BUFFER {
  bool dirty;
  uint8_t buf[BUFFSIZE];
};

volatile BUFFER ramBuffer[BUFFCNT];

volatile uint8_t activeBuf=0;
volatile uint16_t activeOfs=0;
volatile bool overRun=false;

void store(void){
  ramBuffer[activeBuf].buf[activeOfs++]=PINF;
  if(activeOfs>=BUFFSIZE){
    activeOfs = 0;
    ramBuffer[activeBuf++].dirty=true;
    activeBuf %= BUFFCNT;
    if(ramBuffer[activeBuf].dirty){ // overrunning buffer die!
      overRun= true;
      detachInterrupt(2);// too fast, die
      }
    }
}

void blink(uint16_t offTime, uint16_t onTime){
digitalWrite(LED_BUILTIN,HIGH);
delay(onTime);
digitalWrite(LED_BUILTIN,LOW);
delay(offTime);
}


void setup(){
  pinMode(stopPin,INPUT_PULLUP);  // ground pin to log, open to stop logging
  pinMode(LED_BUILTIN,OUTPUT);
  digitalWrite(LED_BUILTIN,LOW); // no error
  Serial.begin(9600);
  for(uint8_t i=0;i<BUFFCNT;i++){
    ramBuffer[i].dirty=false;
  }
  // open file here
  for(uint8_t i=5;i>0;i--){blink(100*i,200*i);} 
  attachInterrupt(2,store,RISING);
}
static uint8_t nextBuf=0;
static unsigned long numBuffersWritten=0;

void flushbuffer(){
if(ramBuffer[nextBuf].dirty){ //check to see if buffer is filled, flush it, mark it clear.
  // File.blockWrite(ramBuffer[nextBuf].buf,BUFFSIZE); // you might have to code a blockwrite function,
  nextBuf = (nextBuf+1)%BUFFCNT; // next buffer in order.
  numBuffersWritten++;
  Serial.print('+'); // tic each written buffer
  if(nextBuf==0) Serial.println();
  }
}

void loop(){
flushbuffer(); // flush buffer to SD if dirty, inc to next buffer
if(overRun||digitalRead(stopPin)){ // Stop Logging
  uint8_t i;
  while(ramBuffer[nextBuf].dirty) flushbuffer(); // save all captured data
  
  // file.close();
  Serial.print(F("\nCaptured "));Serial.print(numBuffersWritten,DEC); Serial.println(F(" buffers"));
  Serial.println(F("closed file "));
  if(overRun) Serial.print(F(" OverRan buffers "));
  while(overRun) { // filled all buffers before the SD write cleared a buffer!
    for(i=0;i<3;i++)blink(50,100);  // S
    for(i=0;i<3;i++)blink(50,300);  // O
    for(i=0;i<3;i++)blink(50,100);  // S
    delay(250);
    }
  }
}

you will have to add the file access routines

chuck.

4

Thanks PaulS,

Which version of the SD library takes 4 arguments for the begin() method?

Adafruit has made some changes to SD library (See here) so you could use pins 10-13 on MEGA for data logging, but now I'm using 50-53 to be sure that their library is not causing any other delays in writing

If one file exists, delete it. Then, open or create a different file. Hmmm...

My stupidiy, never paid attention to that.

Nothing you can do about it on a low-memory, single processor system.

So I should give up?

No, the main question is where is the code you are actually running? The secondary question is how many samples per second you are actually getting? The tertiary question is how do you know that?

  1. I've run it on UNO, MEGA and DUE. Also the SD card that I use is Transcend SDHC Class 10 like this one
  2. The best performance was achieved at 1000 bytes per second
  3. Simply nothing gets written to a file in an SD card
5

Thanks chucktodd,

Currently I'm adding those routines, will read your code several times so I get the whole idea. Then I will test it with several speeds and see if I can save about 10 MB of data at 5000 bytes per second.

Thanks again for your effort!

6

chucktodd,

Using your idea with buffers, we now could write at 10 kHz.

Thanks a lot, you saved my day.

7

I read the fat16lib's topic about logging 40 000 samples per second, but he uses timer for logging and I need to use an interrupt.

The timer is not special. You can use any interrupt to read and buffer samples. The main idea is to allocate a large contiguous file and write block buffers in raw SD mode.

I trigger the ADC with the timer but read the data in an ADC done ISR.

8

javid-y:
chucktodd,

Using your idea with buffers, we now could write at 10 kHz.

Thanks a lot, you saved my day.

Your welcome, you might also think about RLL(Run Length Limited) encoding of the data. If the data has continuous blocks of the same values, you could do a first stage compression if the file.write() speed is the limiting factor.

you might also check out Gillham's Arduino Mega implement of a Logic Analyzer. You could use the logic analyzer (PC) component to log the data, then display it captured data. with Gillham's implementation you could capture 7k samples at 4mhz on a stock Mega2560.

I built a 1MB RAM shield and modified his code to sample 55.5k at over 1mhz, page switching the RAM I was able to sample at 250khz and capture 1M samples.

Chuck

9

javid-y:
Thanks chucktodd,

Currently I'm adding those routines, will read your code several times so I get the whole idea. Then I will test it with several speeds and see if I can save about 10 MB of data at 5000 bytes per second.

Thanks again for your effort!

I'm curious, what are you sampling for 33 minutes a 5k per second? Too slow for sound?

Chuck.

10