Go Down

Topic: data logging: 3 kByte/sec serial in, going to microSD (SDFAT) out (Read 2884 times) previous topic - next topic

John Beale

Aug 13, 2011, 01:36 am Last Edit: Aug 13, 2011, 01:51 am by John Beale Reason: 1
I'm using a Sparkfun 16 MHz 5V Arduino Pro Mini with the Adafruit microSD breakout board. I have a serial data stream coming in that I want to record to uSD card, using the SDFAT library.

The serial data logging examples I've seen are usually slow data at 9600 baud (GPS NMEA, etc).  My serial data (115200 baud) is coming in as one packet of 21 bytes every 6.8 msec, meaning a total of 3088 bytes per second. This data streams in continuously for 30 or more seconds at a time, and I'd like to avoid dropping any data.

I've checked out the SDFAT library data logging example "fastLoggerBeta20110802", and it looks like my overall average data rate of 3 kbytes / sec is easily achieved. In fact, my test shows I can record about 22 kbytes/sec without dropping any data. This is running the SPI bus at half speed, which is all that works with my messy wiring currently.

However, looking at the actual SPI transactions to the microSD with a logic analyzer, I see that the data transfer is quite "bursty" and interleaved with various file operations. It looks like the file directory names on the uSD card are occasionally read out, for example.

So, clearly I need to do some buffering between the continuous stream of serial data input and the faster, but bursty SD write operations.  Just wanted to check,  if a buffered serial-in, SD-out data logger sketch is already available?

robtillaart

writing in larger chuncks reduces overhead a lot. Ideal you write a sector (e.g 512 bytes) at once.
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

John Beale

Thanks for the quick response!  ...but I'm not sure how the file write operations available in SD or SDFAT libraries correspond to the low-level write operations on the SD card.  For example, are the first 512 bytes written to a file necessarily aligned with the SD card's block?  Is there some example code somewhere to demonstrate writing in 512-byte blocks for lower latency?

fat16lib

#3
Aug 13, 2011, 03:47 pm Last Edit: Aug 13, 2011, 04:01 pm by fat16lib Reason: 1
I would like to add an example of this type of Serial logger to SdFat.  I have included a first and almost untested start below.

You don't need to write large chunks in SdFat.  about 20 bytes per write is fine at your data rate.

I did a test setting buffer size to twenty in the SdFat bench.pde example and got the following result:

Quote
Type any character to start
Free RAM: 1137
Type is FAT16
File size 5MB
Starting write test.  Please wait up to a minute
Write 138.58 KB/sec
Maximum latency: 141240 usec, Avg Latency: 139 usec

Starting read test.  Please wait up to a minute
Read 204.18 KB/sec
Maximum latency: 2804 usec, Avg Latency: 93 usec


Note the very high max latency.

The problem is that SD cards have occasional long write latencies so you need more buffering in HardwareSerial.cpp.

In the current version of the Arduino IDE this is easy.  To make the input buffer larger you edit this line

Code: [Select]
 #define RX_BUFFER_SIZE 128

Changing the 128 to 600 allows a write latency of 200 ms at your data rate.

In Arduino 1.0 it will not be so easy.  An output buffer has been added and there is a single buffer size parameter so you can't easily get enough buffer on a 328 Arduino without a larger hack.

Here is a starting point for a Serial logger.  Contact me with a PM if you are interested in developing this example for your use.
Code: [Select]

// Serial data logger example
//
// To achive maximum speed, increase the size of the Serial buffer.
// Edit \hardware\arduino\cores\arduino\HardwareSerial.cpp
// Make the buffer as large as possible in this line for Arduino 0022
// #define RX_BUFFER_SIZE 128
// In Adruino 1.0 edit this line
// #define SERIAL_BUFFER_SIZE 64
//
#include <SdFat.h>
SdFat sd;
SdFile file;

// Idle time for sync call.  If Serial is idle for this time call sync.
const uint32_t IDLE_SYNC_TIME_MSEC = 1000;

// Maximum time between sync() calls in milliseconds.  If MAX_SYNC_TIME_MSEC is
// set to zero, you must provide a way to stop the program and close the file.
// You can also stop the serial stream for at least IDLE_SYNC_TIME_MSEC to
// flush the last data to the file.
const uint32_t MAX_SYNC_TIME_MSEC = 1000;
//------------------------------------------------------------------------------
// pin for error LED
const uint8_t ERROR_LED_PIN = 3;

// Error codes repeat as errno short blinks with a delay between codes.
const uint8_t ERROR_INIT  = 1;  // SD init error
const uint8_t ERROR_OPEN  = 2;  // file open error
const uint8_t ERROR_WRITE = 3;  // write error
const uint8_t ERROR_SYNC  = 4;  // sync error
void errorBlink(uint8_t errno) {
 uint8_t i;
 while (1) {
   for (i = 0; i < errno; i++) {
     digitalWrite(ERROR_LED_PIN, HIGH);
     delay(200);
     digitalWrite(ERROR_LED_PIN, LOW);
     delay(200);
   }
   delay(1600);
 }
}
//------------------------------------------------------------------------------
void setup() {
 Serial.begin(115200);
 pinMode(ERROR_LED_PIN, OUTPUT);
 if (!sd.init()) {
   errorBlink(ERROR_INIT);
 }
 if (!file.open("LOGFILE.BIN", O_WRITE | O_CREAT | O_APPEND)) {
   errorBlink(ERROR_OPEN);
 }
}
//------------------------------------------------------------------------------
// cluster for last sync
uint32_t syncCluster = 0;

// time of last sync
uint32_t syncTime = 0;

// write time
uint32_t writeTime = 0;

uint8_t buf[21];
void loop() {
 uint8_t n = Serial.available();
 if (n > sizeof(buf)) {
   n = sizeof(buf);
 } else if (n == 0) {
   if ((millis() - writeTime) < IDLE_SYNC_TIME_MSEC) return;
   if (!file.sync()) errorBlink(ERROR_SYNC);
   return;
 }
 for (uint8_t i = 0; i < n; i++) {
   buf[i] = Serial.read();
 }
 if (file.write(buf, n) != n) {
   errorBlink(ERROR_WRITE);
 }
 writeTime = millis();
 // never sync if zero
 if (MAX_SYNC_TIME_MSEC == 0) return;
 
 if (syncCluster == file.curCluster()
   && (millis() - syncTime) < MAX_SYNC_TIME_MSEC) return;
   
 if (!file.sync()) {
   errorBlink(ERROR_SYNC);
 }
 syncCluster = file.curCluster();
 syncTime = millis();  
}


John Beale

#4
Aug 15, 2011, 11:55 pm Last Edit: Aug 16, 2011, 12:00 am by John Beale Reason: 1
Thank you so much fat16lib for coding up an example so quickly!  I made the RX_BUFFER_SIZE 128 change to HardwareSerial.cpp as suggested, and made just a few changes to your code. Below is the version that's working for me- I added a few comments to myself, I think the only real change I made is to drop down to half speed on the SPI bus, given my poor wiring configuration at the moment.

This code does "mostly" work for me.  Randomly, maybe once every few thousand lines, I get some dropped characters. Right now my packet format contains six comma-separated hex values, and below is an example showing one too-long line due to dropped characters (line #7524 from my log file, in this case). It seems every "so often" there is a long pause for writing.  Perhaps my uSD card is just too slow, I have a faster one on order.  Also, I guess I should try formatting it first; this card already had some other files on it.

Code: [Select]

Sample data from recorded file:
[...]
41,2B,29,29,158,0
3E,30,25,27,154,0
3F,2F,25,28,15E,0
3F,30,2D,28,155,0
3F,32,25,29,15C,0
43,35,26,28,154,0
3E,32,24,29,15D,0
3E,39,2C,2A,154,0
3F,2D,2B3F,33,25,28,15B,0
40,32,2B,2C,15E,0
3F,32,25,28,155,0
[...]


Code: [Select]
// Serial data logger example
//
// To achive maximum speed, increase the size of the Serial buffer.
// Edit \hardware\arduino\cores\arduino\HardwareSerial.cpp
// Make the buffer as large as possible in this line for Arduino 0022
// #define RX_BUFFER_SIZE 128
// In Arduino 1.0 edit this line
// #define SERIAL_BUFFER_SIZE 64
//
#include <SdFat.h>
SdFat sd;
SdFile file;

const uint8_t SdChipSelect = SS_PIN;  // default is Arduino digital pin 10

// Idle time for sync call.  If Serial is idle for this time call sync.
const uint32_t IDLE_SYNC_TIME_MSEC = 1000;

// Maximum time between sync() calls in milliseconds.  If MAX_SYNC_TIME_MSEC is
// set to zero, you must provide a way to stop the program and close the file.
// You can also stop the serial stream for at least IDLE_SYNC_TIME_MSEC to
// flush the last data to the file.
const uint32_t MAX_SYNC_TIME_MSEC = 1000;
//------------------------------------------------------------------------------
// pin for error LED
const uint8_t ERROR_LED_PIN = 3;

// Error codes repeat as errno short blinks with a delay between codes.
const uint8_t ERROR_INIT  = 1;  // SD init error
const uint8_t ERROR_OPEN  = 2;  // file open error
const uint8_t ERROR_WRITE = 3;  // write error
const uint8_t ERROR_SYNC  = 4;  // sync error
void errorBlink(uint8_t errno) {
 uint8_t i;
 while (1) {
   for (i = 0; i < errno; i++) {
     digitalWrite(ERROR_LED_PIN, HIGH);
     delay(200);
     digitalWrite(ERROR_LED_PIN, LOW);
     delay(200);
   }
   delay(1600);
 }
}
//------------------------------------------------------------------------------
void setup() {
 Serial.begin(115200);            // highest working serial port speed
 
 pinMode(ERROR_LED_PIN, OUTPUT);  // error signal output on this pin
 // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
 // breadboards.  use SPI_FULL_SPEED for better performance.
 if (!sd.init(SPI_HALF_SPEED, SdChipSelect)) {
   errorBlink(ERROR_INIT);
 }
 
 if (!file.open("SERLOG.CSV", O_WRITE | O_CREAT | O_APPEND)) {
   errorBlink(ERROR_OPEN);
 }
}
//------------------------------------------------------------------------------
// cluster for last sync
uint32_t syncCluster = 0;

// time of last sync
uint32_t syncTime = 0;

// write time
uint32_t writeTime = 0;

uint8_t buf[26];    // longest expected serial packet
void loop() {
 uint8_t n = Serial.available();
 if (n > sizeof(buf)) {
   n = sizeof(buf);
 } else if (n == 0) {  // no serial characters to handle... Is it time to SYNC yet?
   if ((millis() - writeTime) < IDLE_SYNC_TIME_MSEC) return;
   if (!file.sync()) errorBlink(ERROR_SYNC);
   return;
 }
 for (uint8_t i = 0; i < n; i++) { // load n serial characters into our write buffer
   buf[i] = Serial.read();
 }
 if (file.write(buf, n) != n) {  // ...and write to SD card
   errorBlink(ERROR_WRITE);
 }
 writeTime = millis();
 
 // never sync if zero
 if (MAX_SYNC_TIME_MSEC == 0) return;
 
 if (syncCluster == file.curCluster()
   && (millis() - syncTime) < MAX_SYNC_TIME_MSEC) return;
   
 if (!file.sync()) {
   errorBlink(ERROR_SYNC);
 }
 syncCluster = file.curCluster();
 syncTime = millis();  
}

John Beale

Just as a followup, I got the SD/MCC adapter board from www.gravitech.us and used it with a Transcend 16GB SDHC Class 10 card. With this I was able to log over 4 hours of data at 7 kB/sec with no dropped characters, which is good enough for now.  I used the SdFat "AnalogLogger" example with minor modifications (mostly just increased the data written to the card, write one line of data every 6.2 msec, and call logfile.flush() every second.)

With the basic 2GB Sandisk microSD cards I had been using, I could not go even one minute at 3.5 kB/sec without some dropped characters.

Code: [Select]
#define FLUSH_INTERVAL  1000  // msec between SD file flush
...
  logfile << buf;   // log data

  if ((m - lastflush) > FLUSH_INTERVAL) {
    logfile.flush();    // flush data to SD
    lastflush = m;
    cout << buf;      // send data to serial port
  }

Go Up