Improving SdCard Write Time

[MorganS]

On most Arduinos, yield() is a virtual function. It is empty and just waiting for you to write a "real" one to do what you need during yields.

It originated on the Arduino Due, with the scheduler library. That yields during any delay() to let other scheduled tasks run. It was such a good idea that it's now on all Arduinos.

Think about servicing the WiFi side of the chip during a delay(). That is what it is for.

[kinser86]

Ok there is a lot of information here I need to digest in the previous post but first let me address the change in SdCard.

Per SpaceJohn, I changed to a different card I had which is a Kingston 8GB SDC10/8GB. I formatted the card with the sdcard.org program and then ran the SdInfo.ino program. The init time is substantially lower (from 76ms to 3ms).

Code: [Select]


init time: 3 ms

Card type: SDHC

Manufacturer ID: 0X41
OEM ID: 42
Product: SD8GB
Version: 3.0
Serial number: 0XC3045500
Manufacturing date: 1/2015

cardSize: 7969.18 MB (MB = 1,000,000 bytes)
flashEraseSize: 128 blocks
eraseSingleBlock: true
OCR: 0XC0FF8000

SD Partition Table
part,boot,type,start,length
1,0X0,0XB,8192,15556608
2,0X0,0X0,0,0
3,0X0,0X0,0,0
4,0X0,0X0,0,0

Volume is FAT32
blocksPerCluster: 64
clusterCount: 242944
freeClusters: 242941
freeSpace: 7960.69 MB (MB = 1,000,000 bytes)
fatStartBlock: 12586
fatCount: 2
blocksPerFat: 1899
rootDirStart: 2
dataStartBlock: 16384


The +200ms blips in my data are now in the 100ms range. I did a few short test and the results were as follows:

• 33,000 samples (10 samples greater than 50-ms)
• 7,100 samples (1 sample greater than 50-ms)
• 8,700 samples (0 samples greater than 50-ms)

Is there an oppourtunity in my current code in the _writeState with respect to checking the length of the buffer?

if (buffer.length() >= 312) {

As I mentioned before, it is maxing out at 463. I have noticed that the program performs better (fewer instances of +50ms) when the size is larger, and worse (more instances of +50ms) when it is smaller.

A general question, am I exceeding the capabilities of a FSM in this scenario? I bet there are endless ways of writing this program and ultimately for me this was an exercise for me in learning FSM. Before I go buying FRAM breakouts and spending money on hardware, am I at a point with this program where I should be writing binary data or am I still within reason to have readable data at 20hz?

I will keep reading the previous posts so I can digest the information some more. Thank you everyone for your input this is great help so far.

[MorganS]

What FSM? Your code has no states.

Yes there is a variable called state but it is not being used in a FSM.

[kinser86]

What FSM? Your code has no states.

Yes there is a variable called state but it is not being used in a FSM.

Maybe phrasing the question this way would be better.

Am I limited with a switch case structure?

[ShermanP]

I don't know enough about C++ to comment on your switch case code.  But I would be curious about what would happen if you did away with the buffer entirely and simply wrote the successive values to the file, and let SdFat handle the buffering.  And only flush and close the file when you want to stop logging.

[MorganS]

What FSM? Your code has no states.

I wrote that on a phone and I wanted to clarify it a bit more. The code has no states because it has no/very few break; statements in the switch(). Because of that it just simply works its way down the list as if the switch() wasn't there. It appears to work because it is a very linear state machine and never attempts to stay in one state or branch to different states.

I don't know enough about C++ to comment on your switch case code.  But I would be curious about what would happen if you did away with the buffer entirely and simply wrote the successive values to the file, and let SdFat handle the buffering.  And only flush and close the file when you want to stop logging.

Two things happen:

1. You are at the mercy of the library and card. Whenever it decides to stop and think for 20 milliseconds, you are stuck. You cannot do anything. Your code is not executed.

2. If you ever lose power then the entire file is lost. Without calling flush() or sync() or close() then it never writes the actual length of the file to the directory and the PC sees an empty file.

[ShermanP]

Two things happen:

1. You are at the mercy of the library and card. Whenever it decides to stop and think for 20 milliseconds, you are stuck. You cannot do anything. Your code is not executed.

2. If you ever lose power then the entire file is lost. Without calling flush() or sync() or close() then it never writes the actual length of the file to the directory and the PC sees an empty file.

1.  I've already made my pitch for an interrupt driven process using two buffers.  But his current code doesn't do anything to get around either library or card slowdowns.  He can batch things up into a buffer all he wants, but in the end he's sending the buffer contents to the library code, which is going to convert it to 512-byte batches to send to the card.  So it just seems to me that using the buffer this way only takes up ram, and doesn't really accomplish anything.

2.  I understand about the directory entry.  I think the FAT table also has to be updated, although I guess that happens as additional clusters are written to.   But if the risk of power loss isn't significant, doing away with the closing and reopening overhead, combined with using a fast card, might let him get this done without losing any readings.  Anyway, it might be interesting just to see whether it makes a difference or not.

[SpaceJohn]

ShermanP,
I just use my Windows 7 formatting routine to format my SDHC cards. When I did Assembly programming, EPROMS & EEPROMS were set to "FF" as the erased state. I assume that FLASH, following EEPROM, applies the same status. I read that the SD cards, =<2 GB, require blocks of memory to be erased before programming, which inhibited data transfer onto the SD card. It appears, from my limited work with SDHC cards, that the SDHC & SDXC cards erase on the fly. I need to verify this.
Having a time stamp header with my data reduces the necessity for precise time interval captures. Sometimes it is necessary.  Reviewing data captured every 2 milliseconds as a rocket penetrates the ground on impact is very exciting and necessary. It expanded an 18ms event into a time scale I could study. Having 3 data points or 9 data points captured with a known time stamp was all that was necessary in the final analysis. Yes, the Arduino system survived a 716 G impact with no damage. Can't say the same for the battery.

If the project requires microsecond precision data then the interval time must be precise. External buffers are then a requirement, but they also open you to significant data loss with power interruption or other damage.

The TI MSP430 is interesting. For many years I worked with RAMTRON placing FRAMs in control products used in the Semi Industry. The RAMTRON rep visited me often and supplied as many samples as I needed. I love FRAMS.

I put together a simple 3 channel datalogger running at 20 Hz using only SD and SPI library calls and transferring data in CSV String arrays. (The current SD.h library is running FAT.) The average interval time was 50.75 ms. Interval times ranged from 48 to 56 ms, weighted to the plus 50 ms side. I recorded zero intervals above 56 ms until I purposely started overflowing the SDHC internal buffer, sporadic interval times then jumped  to 99 ms. Data recorded on a Verbatum 16GB Class 10 SDHC.
 
One issue I have with the Arduino Sketch Libraries being Open Source. Once I develop a project program, I must lock it down to the specific editor and libraries used. Updates to libraries or addition of other component or processor libraries has affected throughput to SD cards.                

[kinser86]

I have gone back through my code and re-evaluated the use of buffering and ended up removing it completely. I have also removed the frequent flush() operations, reducing the close() operation to only occur once. The risk of power loss in my situation will be low since the start/stop is controlled by a person and it will not be on a battery supply. I have noticed an improvement by removing my previous buffering. I performed a couple of test and here are the results:

•   1,000 samples (0 greater than 50ms)
•   7,000 samples (0 greater than 50ms)
• 13,000 samples (1 sample at 108ms at the 65th sample)
• 48,000 samples (6 samples around 110ms towards the middle of the dataset)

I no longer see +200ms wait times, rather +100ms wait times closer to 100ms. I feel like this is helping quite a bit, much better than what I saw prior to posting my question.

I did order some other SdCards to try and see if it will help anymore but they haven't come in yet.

Code: [Select]


// Libraries
#include <SdFat.h>
#include <RTClib.h>

RTC_PCF8523 rtc;

enum state {
  _readState,     // Read analog values
  _displayState,  // Display values (not yet programmed in)
  _createState,   // Create a file for datalogging
  _writeState,    // Write data to file
  _errorState     // Stop the program if something is wrong
};

state _currentState;

// Declare Variables
struct sample {
  char* _name;
  const int pin;      // Input pin
  unsigned int raw;   // Raw value
  float value;        // Calculated value
};

sample Val1 = {"Val1", 2, 0, 0};   // Voltage range 0.5 - 4.5 VDC
sample Val2 = {"Val2", 3, 0, 0};    // Voltage range 0.0 - 5.0 VDC

const int ledRecord = 6;
const int pinSwitch = 7;
const int ledRead =  8;
const int ledError = 9;
const int pinSD = 10;

unsigned long currentMillis;  // Used for storing the latest time
unsigned long previousMillis = 0;      // Store the last time the program ran
const unsigned long interval = 50;     // Sample frequency (milliseconds)

int pinState;

String filename;
char _header[20];
SdFat SD;
File dataFile;    // SD Card

void setup() {
  // Start serial
  Serial.begin(57600);
  while (!Serial);
  pinMode(ledError, OUTPUT);
  pinMode(pinSwitch, INPUT_PULLUP);
  pinMode(ledRecord, OUTPUT);
  pinMode(ledRead, OUTPUT);
  pinMode(pinSD, OUTPUT);
  // Start RTC
  Serial.print("\n\nStarting RTC...");
  rtc.begin();
  // Show all lights as part of boot sequence
  digitalWrite(ledRecord, HIGH);
  digitalWrite(ledRead, HIGH);
  digitalWrite(ledError, HIGH);
  // Throw away analog read
  pinState = digitalRead(pinSwitch);
  Val1.raw = analogRead(Val1.pin);
  Val2.raw = analogRead(Val2.pin);
  delay(500);
  digitalWrite(ledRecord, LOW);
  digitalWrite(ledRead, LOW);
  digitalWrite(ledError, LOW);
  Serial.println(" RTC started!");
  Serial.print("Initializing SD card...");
  if (!SD.begin(pinSD)) {
    Serial.println("Card failed, or not present");
    digitalWrite(ledError, HIGH);
    _currentState = _errorState;
    return;
  }
  Serial.println(" SD card initialized!");
  // Test Sd Card before continuing
  // Create header for files
  sprintf(_header, "Time(ms),%s,%s",Val1._name,Val2._name);
  // Print header for serial logging
  filename = "TEST.txt";
  dataFile = SD.open(filename, O_WRITE | O_CREAT);
  dataFile.println(_header);
  dataFile.println("Test");
  if (!dataFile){
    Serial.println("Unable to write to Sd Card.");
    digitalWrite(ledError, HIGH);
    _currentState = _errorState;
    return;
  }
  dataFile.remove();
  filename = "";
  Serial.println("Program ready.");
}

void loop() {
  pinState = digitalRead(pinSwitch);
  digitalWrite(ledRead, HIGH);
  digitalWrite(ledRecord, LOW);
  currentMillis = millis();

  if ((currentMillis - previousMillis) >= interval) {
    switch (_currentState) {

      case _readState:
        _currentState = _readState;
        digitalWrite(ledRead, LOW);   // Green LED OFF
        //Read Values
        Val1.raw = analogRead(Val1.pin);
        Val2.raw = analogRead(Val2.pin);

      case _displayState:
        _currentState = _displayState;
        previousMillis = currentMillis;
        if (pinState != LOW) {
          if (filename != ""){
            Serial.print("Closing file...");
            dataFile.close();
            filename = "";
            Serial.println("file closed.");
          }
          _currentState = _readState;
          break;
        };

      case _createState:
        _currentState = _createState;
        if (filename == "") {
          DateTime now = rtc.now();
          filename = String(now.unixtime(), DEC);
          filename = filename + ".txt";
          Serial.print(filename);
          Serial.println(" created!");
          dataFile = SD.open(filename, O_CREAT | O_APPEND | O_WRITE);     // Open file
          dataFile.println(_header);
        };

      case _writeState:
        _currentState = _writeState;
        digitalWrite(ledRecord, HIGH);
        dataFile.print(millis());
        dataFile.print(",");
        dataFile.print(Val1.raw);
        dataFile.print(",");
        dataFile.println(Val2.raw);
        _currentState = _readState;
        previousMillis = currentMillis;
        break;

      case _errorState:
        Serial.println("Current State: _errorState");
        digitalWrite(ledError, HIGH);
        _currentState = _errorState;
    }
  }
}


I put together a simple 3 channel datalogger running at 20 Hz using only SD and SPI library calls and transferring data in CSV String arrays. (The current SD.h library is running FAT.) The average interval time was 50.75 ms. Interval times ranged from 48 to 56 ms, weighted to the plus 50 ms side. I recorded zero intervals above 56 ms until I purposely started overflowing the SDHC internal buffer, sporadic interval times then jumped  to 99 ms. Data recorded on a Verbatum 16GB Class 10 SDHC.

I hope to see interval times within this range (48-56ms). Hopefully a different SdCard will help.

[ShermanP]

Spacejohn,
I'm not sure how Windows formats an SD card, in particular whether it erases the card or not.  I have the same question about the SD Formatter.  There are low-level SD commands, including CMD32, CMD33 and CMD38, which provide for erasing a sequence of sectors, but you can't tell by looking whether that has been done.  If I get time, I will try to write something that I know erases the entire card.  After doing that, you would use the SD Formatter to do a quick format to set up the partition and file structure.

kinser86,
It looks like you've pretty much got this done.  In the end, you're going to be at the mercy of the SD card controller's firmware, as well as the skill of whoever wrote the Arduino FAT libraries.  So logging directly to an SD card may not be the ideal method for fast data logging. But it's cheap, and seems to work reasonably well up to a point.

[kinser86]

I got my 16gb SdCard in and formatted it using the SdFat formatting program in the Examples. I no longer see the long wait times, rather an a swing of +/- 6ms with an average of 50ms. This will definitely work for what I need. I appreciate everyone's guidance and support. As ShermanP mentioned, I am at the mercy of the library and it is working well enough for me at the moment to no longer push the envelope.  Thanks again everyone!