Reducing time spent wirting to SD while data logging to minimize missed pulses

Hi,

I’m working on a fuel flow datalogger and I have a question regarding the programming.

I am using the 84 MHz DUE and 4 input channels where I count the pulses in one second using interrupts on all 4 inputs.
I built a box with an SDwrite indicator LED and a reset button. See pictures.

It counts pulses during 1000 ms and calculates instantaneous flow rate as: 1/K3600(number of pulses) [the number of pulses=frequency]
there is a supply and return line, as some of the fuel cools the injectors and returns to the tank. (S & R)
Burn rate is calculated as S-R and Total burn, T, is calculated as the accumulated burn for each second.

When 1000 ms have passed, it calculates the flow rates and other values that are to be logged, write them to SD card and prints to a 20x4 LCD.

As I understand it, while the SD is writing, the interrupt can only set a single flag if an event has occurred, meaning it can only count a single pulse for each channel during this time. The write speed of the SD therefore surely will affect the amount of missed pulses.

I have a total burn variable, which would be the one parameter suffering the most from this as the pulses during SD writing is lost.

Is there a way to count every single pulse while writing to SD? Also I would like to also only write every 10 seconds to the SD card to minimize wear and improve card lifetime. How would I go about this? I would think that storing data for 1 seconds in a bunch of variables and then writing them all at once.

See my loop here:

void loop(){
  detachInterrupt(0);
  detachInterrupt(1);
  detachInterrupt(4);
  detachInterrupt(5);
  
  
  if(millis()-startTime>1000UL){      // Every 1000 ms, flow rates are calculated and written
    
    // Calculates flow rates - 0.001051525 is equal to 1/951 (pulsevolume) and multiplying is faster than dividing
    
  double flow1 = 1/951*3600*pulseCount1;   // [L/h]  K-factor=951 pulses per liter
  double flow2 =1/951*3600*pulseCount2;   // [L/h]
  double flow3 = 1/951*3600*pulseCount3;   // [L/h]
  double flow4 = 1/951*3600*pulseCount4;   // [L/h]
  
  // calculates total burn during the last second
  double accumulated1 = (pulseCount1 - pulseCount2) * pulsevolume;
  double accumulated2 = (pulseCount3 - pulseCount4) * pulsevolume;
  
  // Updates Total accumulated burn over time
  total1 = total1 + accumulated1;
  total2 = total2 + accumulated2;  
    
    //--------------Writes values to SD card--------------
    File dataFile = SD.open("datalog.csv", FILE_WRITE);

       // if the file is available, write to it:
       if (dataFile) {
        digitalWrite(1, HIGH); // Signals writing to SD
        DateTime now = rtc.now();  
        dataFile.print(now.hour(), DEC);
        dataFile.print(':');
        dataFile.print(now.minute(), DEC);
        dataFile.print(':');
        dataFile.print(now.second(), DEC);
        dataFile.print(",");
        dataFile.print(flow1);
        dataFile.print(",");
        dataFile.print(flow2);
        dataFile.print(",");
        dataFile.print(flow1-flow2);
        dataFile.print(",");
        dataFile.print(total1);
        dataFile.print(",");
        dataFile.print(flow3);
        dataFile.print(",");
        dataFile.print(flow4);
        dataFile.print(",");
        dataFile.print(flow3-flow4);
        dataFile.print(",");
        dataFile.println(total2);
        dataFile.close();
        digitalWrite(1, LOW); //SD write blink signal off
        
        //----------------- Updates values on LCD -----------------
    lcd.setCursor(9,0);
    lcd.print(" ");
    lcd.setCursor(3,0);
    lcd.print(flow1);
    
    lcd.setCursor(9,1);
    lcd.print(" ");
    lcd.setCursor(3,1);
    lcd.print(flow2);
    
    lcd.setCursor(9,2);
    lcd.print(" ");
    lcd.setCursor(3,2);
    lcd.print(flow3);
    
    lcd.setCursor(9,3);
    lcd.print(" ");
    lcd.setCursor(3,3);
    lcd.print(flow4);
    
    lcd.setCursor(19,0);
    lcd.print(" ");
    lcd.setCursor(13,0);
    lcd.print(flow1-flow2);
    
    lcd.setCursor(19,1);
    lcd.print(" ");
    lcd.setCursor(13,1);
    lcd.print(total1);
    
    lcd.setCursor(19,2);
    lcd.print(" ");
    lcd.setCursor(13,2);
    lcd.print(flow3-flow4);
    
    lcd.setCursor(19,3);
    lcd.print(" ");
    lcd.setCursor(13,3);
    lcd.print(total2);
    
    // --------------resets pulseCount variables--------------
    pulseCount1 = 0;
    pulseCount2 = 0;
    pulseCount3 = 0;
    pulseCount4 = 0;
    startTime = millis();
  }
  }
  // ---------Re-activates interrupts-----------------
  attachInterrupt(0, pulseCounter1, FALLING);
  attachInterrupt(1, pulseCounter2, FALLING);
  attachInterrupt(4, pulseCounter3, FALLING);
  attachInterrupt(5, pulseCounter4, FALLING);
}

As I understand it, while the SD is writing, the interrupt can only set a single flag if an event has occurred, meaning it can only count a single pulse for each channel during this time. The write speed of the SD therefore surely will affect the amount of missed pulses.

You are assuming that interrupts are disabled while writing to the SD card. I'm not sure that this is a valid assumption.

I'm not sure whether you have a problem, and obviously you would want to confirm that before you go any further by proving that the SD write takes longer than a single interrupt period and that you lose interrupts during the write. I've seen examples logging analog values to SD at tens of KHz so I suspect either there's no problem, or there is a known solution to it that you need to find (the examples have been referenced in the forum - I think it should be possible to find them).

Worst case if you can't find any other solution would be to shove the logged data out over a serial port to an ArduLog which can write to the SD for you. Shoving data out over the serial port certainly doesn't cause any conflict since the serial interrupt is very short.

Hi PaulS and PeterH,

PaulS: Interrupts are detached during SD writing. That would make my assumption correct, right? Please ellaborate.

PeterH: I'm not sure either... I will say that I have 4 channels with a max frequency of 500 Hz, and I set an LED to blink when writing to the SD card and my slowest is on for at least 0,2 seconds or so... Th ciard write speed would be a way to minimize write time and thereby missed pulses, but if there is a way to program around it, that would be awesome.

Which SD library are you using ?

I just took a look in the Adafruit one on GitHub and it does disable interrupts in at least one place

i.e in sd2card.cpp

/** Receive a byte from the card */
  static  uint8_t spiRec(void) {
  if (clockPin_ == -1) {
    #ifndef USE_SPI_LIB
      spiSend(0XFF);
      return SPDR;
    #else
      return SPI.transfer(0xFF);
    #endif
  } else {
    uint8_t data = 0;
    // no interrupts during byte receive - about 8 us
    noInterrupts();
    // output pin high - like sending 0XFF
    *mosiport |= mosipinmask;
    
    for (uint8_t i = 0; i < 8; i++) {
      *clkport |=  clkpinmask;
      data <<= 1;
      
      //if (fastDigitalRead(SPI_MISO_PIN)) data |= 1;
      if ((*misoport) & misopinmask)  data |= 1;
      
      *clkport &=  ~clkpinmask;
      
      // adjust so SCK is nice
      nop;
      nop;
    }
    // enable interrupts
    interrupts();
    return data;
    } 
  }

I presume this code is being called ?? And if so, yes it would disable interrupts.

However there may be ways to get around this, by modifying the SD library.

I don’t mean remove the noInterrups() statements, but if your pulse stream is fairly regular over a short period of time, you may be able to synchronise the writing of the SD with your interrups, so you can guarantee that you will not get any interrupts during this period.

Or… Offload the SD writing onto a second Arduino which you comminicate with, either over Serial, SPI or I2C.
i.e just buy a Ardunio mini and an micro SD card reader (total cost < $10) and connect it to your existing system.

This article at openbci describes how to access the SD card at a low level to pre-allocate the file storage and make full use of the SD card cache. It reports a write time of around 700 us to write a block of 512 bytes, compared to 20 ms or more using the standard library. If you only need to receive data at 500Hz or so, this is probably fast enough to be in the right ballpark, bearing in mind that yu would only incur this write time once per block of data, not once per sample.

I am using SD card logging but time is not critical. I'm measuring water flow through a flume. On the other hand, I found that formatting the SD card with your PC's default formatting function dramatically reduces its speed. Try the Sd formatter from Sd association and use SdFat library instead. The difference was between "no, this won't work" and "yeah!". I think the only reason adafruit has an SD library is to use software SPI on their data logging shields. It is missing ICSP header so your DUE is probably bit banging away at low speed. If you switch to SdFat library and wire the pins 10-13 to the ICSP (or whatever pins DUE has for SPI) it would use hardware SPI and become faster (I guess).

@RogerClark

I'm using the Adafruit library, so I guess interrupts are disabled at some point. Thanks for pointing that out. Yes, I also considered a second arduino for doing the logging part. As PeterH suggested, the ArduLog seems lige a nice little board for this purpose.

@PeterH

I'll read that article, I want to write as little as possible to the SD card, so think I will have to store each sample in a separate string-variable and then write 5 or 10 or how many is suitable at the same time (to minimize wear on the card).

@liudr

I have diferent speed cards, the slowest is an old 32 mb card and that gives a much longer write blink than the other cards I have.
I will say that I think that it is the only card I formatted myself. I am using Ubuntu, don't know if its formatter has the same issue as the Windows one?

Yes, no linux formatter is any better than windows/mac ones. All of them assume they are formatting damn rotary magnetic platters and behave that way. There is no random access and no need to make small allocation blocks on flash, indeed, large allocation blocks that match the embedded NAND and controller's capability is the only way to run them efficiently. After all, SD cards were made to store videos or pictures on a continuous way, with large files, not a bunch of small files and using small blocks. SD card formatting into linux ex system is especially bad so SD based small linux systems like raspberry pi is quite slow to run. If possible, format it! You will compare the difference and thank that formatter program.