SdFat update on GitHub

I have posted a new version of SdFat on GitHub GitHub - greiman/SdFat: Arduino FAT16/FAT32 exFAT Library.

This version of SdFat has a large number of internal modifications. I have done a number of tests but I suspect it may break some applications. Please try this SdFat library and report any problems.

Try the LowLatencyLogger example, it uses one of the new features. It is easily modifiable and does not use an RTOS or tricky ISR. I was able to log four analog pins on an Uno at 500 Hz and at 5000 Hz on a Due. I used a low cost class 4 SanDisk card.

Here are the three functions I used. You need to modify these for your needs.

const uint8_t ADC_DIM = 4;
struct data_t {
  unsigned long time;
  unsigned short adc[ADC_DIM];
};

// Acquire a data record.
void acquireData(data_t* data) {
  data->time = micros();
  for (int i = 0; i < ADC_DIM; i++) {
    data->adc[i] = analogRead(i);
  }
}

// Print a data record.
void printData(Print* pr, data_t* data) {
  pr->print(data->time);
  for (int i = 0; i < ADC_DIM; i++) {
    pr->write(',');  
    pr->print(data->adc[i]);
  }
  pr->println();
}

// Print data header.
void printHeader(Print* pr) {
  pr->print(F("time"));
  for (int i = 0; i < ADC_DIM; i++) {
    pr->print(F(",adc"));
    pr->print(i);
  }
  pr->println();
}

I have added several new examples, deleted a few, and modified many of the rest.

The StdioBench example demonstrates another feature I am working on, stdio style I/O. The StdioStream class has stdio style buffering with a configurable amount of ungetc() push-back to make parsing easier.

Here are results comparing Arduino Print with StdioStream formatting on Uno.

Starting test
uint8_t 0 to 255, 100 times
fileSize: 116500
print millis: 6180
stdio millis: 837
ratio: 7.38

uint16_t 0 to 20000
fileSize: 128890
print millis: 6500
stdio millis: 863
ratio: 7.53

uint32_t 0 to 20000
fileSize: 128890
print millis: 6475
stdio millis: 962
ratio: 6.73

uint32_t 1000000000 to 1000010000
fileSize: 120000
print millis: 5771
stdio millis: 866
ratio: 6.66

float nnn.ffff, 10000 times
fileSize: 100000
print millis: 9366
stdio millis: 1498
ratio: 6.25

I've been using SdFat version 20130710 in a program that logs records at a rate up to 12 KB/sec on an Arduino Fio @ 8MHz. Most timing problems were resolved with a branded micro SD card, which yielded much smaller latency.

The program uses the microcontroller to its limits, both in RAM and ROM, and is running quite stable at a sampling rate of 250 Hz (1 per 4ms). I am using nilRTOS with two threads, the major doing the sampling (I2C, ADC), the minor doing the SD card logging and user interface via serial.

When I read about the new version of the SdFat library, I was hoping to have some benefit updating.

The first thing I noticed was, that my user interface was not responding anymore, once data logging to file had been started. Some investingation lead me to template definition printFileT() in file SdBaseFilePrint.cpp, which is taking much longer than before. The do ... while loop looks suspicous (compared to previous version):

  do {
    Type m = value;
    value /= 10;
    *--str = '0' + m - 10*value;
  } while (value);

This sums up over all my fields being logged. The same logging now takes more that 2ms compared to less than 1.5ms before. Finally, no time remained to listen for user input.

From comments in the code of FmtNumber.cpp I understood that the previous procedure to convert numbers to characters is not correct at all times? Is that the resaon it has been removed (or rather the code that was in printFieldT before)? Note: I excluded this utility module because I didn't have the patience to resolve some compiler errors. Because I do not need to convert any numbers to formatted decimals, it would be nice to have an option that has the previous, better performance.

Besides, I think the SdFat library is a great piece of software for use in programs utilizing SD cards. Thank you for sharing this effort!

I modified SdBaseFile::printField to be correct. I intend to update SdBaseFile::printField to use routines in FmtNumber.cpp now that I believe the bug in the fast format routines are fixed.

The fastest print routines are now in the new StdioStream class. this class uses very fast buffering and FmtNumber.cpp. It will be faster than SdBaseFile::printField even after I modify SdBaseFile::printField to use FmtNumber.cpp.

Look at and run this example https://github.com/greiman/SdFat/blob/master/SdFat/examples/StdioBench/StdioBench.ino.

Thanks for your explanations.

I’ll give the StdioStream class a try when I find some time to do so.

So far I was able to further optimize my own code and get everything working with the new SdFat version as well.

I posted a beta of SdFat with a faster printField() for the SdFile and SdBaseFile classes https://github.com/greiman/SdFat-beta.

I also added a member function for float:

  int printField (float value, char term, uint8_t prec = 2);

printField() is about four times faster than println(). Here are results from the PrintBenchmark example comparing printField() with println():

Test of println(uint16_t) Time 6.73 sec File size 128.89 KB Write 19.15 KB/sec Maximum latency: 5168 usec, Minimum Latency: 172 usec, Avg Latency: 328 usec

Test of printField(uint16_t, char) Time 1.59 sec File size 128.89 KB Write 80.91 KB/sec Maximum latency: 72400 usec, Minimum Latency: 44 usec, Avg Latency: 72 usec

Test of println(uint32_t) Time 9.91 sec File size 200.00 KB Write 20.17 KB/sec Maximum latency: 5532 usec, Minimum Latency: 464 usec, Avg Latency: 488 usec

Test of printField(uint32_t, char) Time 2.07 sec File size 200.00 KB Write 96.67 KB/sec Maximum latency: 73676 usec, Minimum Latency: 72 usec, Avg Latency: 95 usec

Test of println(float) Time 12.88 sec File size 149.00 KB Write 11.57 KB/sec Maximum latency: 5652 usec, Minimum Latency: 548 usec, Avg Latency: 636 usec

Test of printField(float, char) Time 3.41 sec File size 149.00 KB Write 43.64 KB/sec Maximum latency: 5272 usec, Minimum Latency: 136 usec, Avg Latency: 163 usec

Done!

This beta also has software SPI for Teensy 3 and Due. See the SoftwareSPI.txt file.

SPI transactions have been added. See SPI_Transactions.txt.

great work! I implemented it on a Bean http://punchthrough.com/bean/ and everything of the SdFat runs so far exept of the LowLatencyLogger:

as soon at it arrives at while (sd.exists(binName)) in the logData() it jumps back to the main loop and starts over. Do you have an idea by what it might be caused?

n the logData() it jumps back to the main loop and starts over. Do you have an idea by what it might be caused?

It may be a memory problem. LowLatencyLogger uses an extra 512 byte buffer for logData and other functions. This extra buffer allows LowLatencyLogger to avoid SD write latency problems and can't be removed.

There must be about 800-900 bytes of free RAM when LowLatencyLogger starts. I don't know how much RAM is used by Bean's software. On an Uno LowLatencyLogger prints this:

FreeRam: 1148 Records/block: 42

type: c - convert file to CSV d - dump data to Serial e - overrun error details r - record data

Bean looks like a fun device.