How to treat SERIAL INTERRUPTS, while recording to SD CARD on SPI.

Here is my problem… Im “sniffing” a serial bus from a product, and want to record EVERY message that is flowing on that Bus into an SD Card. Basically i READ the serial bus (regular UART serial port on arduino UNO), and every ~200 bytes in a buffer, I record that to the SD Card.

My code was pretty simple, but I then realized I was missing some messages. I did some testing and realized that the SD card write routine was taking sometimes up to 300ms to finish, and in the mean time, I was loosing messages from my serial port. I use the “SERIALEVENT()” routine, but apparently this is only run between each LOOP() time. With that, If Im in the SD writing procedure, which I have on the LOOP(), then if the SD takes longer than say 50ms, my serial buffer will overflow, and I dont execute the SerialEvent() routine to clean it up, and thats how I think I loose messages…

Ideas?

The code below I used to understand the delay to write to the serial port, where I saw the “LONG” and somewhat unpredictable write times to the SD card.

/*
  SD card datalogger
 
 This example shows how to log data from three analog sensors 
 to an SD card using the SD library.
 	
 The circuit:
 * analog sensors on analog ins 0, 1, and 2
 * SD card attached to SPI bus as follows:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 4
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 
 This example code is in the public domain.
 	 
 */

#include <SD.h>

// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.
const int chipSelect = 4;

void setup()
{
 // Open serial communications and wait for port to open:
  Serial.begin(115200);
   while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }


  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("Card initialized.  >>>  /n");
  Serial.println(millis());
  File dataFile;
//  String dataString = "";
   for (long int i = 2; i < 500000; i++)  {
//   dataString += millis();
    
  }
   Serial.println("End of FOR: ");
   Serial.println(millis());
   int i = 0;
}





void loop()
{
  // make a string for assembling the data to log:
  String dataString = "";
  dataString += millis();
// Just created a "big" string , to log to the SD Card
  dataString += ";12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890";

  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  File dataFile = SD.open("datalog2.txt", FILE_WRITE);

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.flush();
    dataFile.close();
  }  
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  } 
  int i = i+1;
}

Moderator edit: CODE TAGS

 int i = i+1;

What's that?

Hi, some left over code.. not used for the particular purpose of the example.

You have to use an FIFO in order to compensate the sdcard's write latencies. I've been using NilRtos with NilFIFO for that purpose - logging up to ~20kB/sec of text data, serial works during logging as well..

Sounds very interesting.. But I couldn't get the solution.. Can u pls share more details about this NilRtos FIFO you mentioned? Maybe some sample code?

Thanks!

Download and install NilRtos: http://forum.arduino.cc/index.php?topic=178532.0

There is an example on data logging, a good point to start. Have a look on the NilRTOS.html documentation there - the fifo principle is explained there along with nice pictures ( see Examples in the html ). Also search forum for Nilrtos topics.

You have to get into the concept yourself, after that it might be easy. You need a periodic task which reads data (from internal/external ADCs for example) and writes the data (in form of a structure) into the fifo (when the fifo is not full), and then you need a second task which reads the fifo (when the fifo is not empty) and writes the data onto the sdcard. The synchronization is done via semaphores.

Thus when the sdcard is not ready for a longer period of time you fills in the fifo, when the card is ready again you writes the fifo content onto the card. Those two tasks are asynchronous - the first task is periodic (ie. 10ms sampling rate) and the second one is with "as fast as possible" timing.. You may print out the fifo statistic during the sampling, you may see how much of fifo the process consumes. You may log the fifo stats (consumer and producer sides) on the card along with each measurement record - you will see how the fifo fills/empties during your logging, based on the sdcard's latency (latency - the actual time the "write of a fifo record into sdcard" took - you can measure it with micros() and log on the card as well).

In parallel you may run many other tasks, for example a third task which does something via Serial.

See also: http://forum.arduino.cc/index.php?topic=144715.30

Here is an example - each csv record is sampled with 4ms period. The last five data are fifo’s consumer/producer watermarks and the write latency. You may see when the sdcard’s latency jumped to 17.472msecs the fifo buffered the records (the “0,149” means the fifo is empty, “4,145” means there are 4 records buffered in). The fifo then wrote the 4 buffered data records (and the new records) within 9 subsequent measurements onto the sdcard. No record was lost. The average time to write a record onto sdcard was ~2msecs. During this logging you may use Serial for communication with the system, as the Serial is also buffered.

,1379355276,104,0.049,85.71,92.24,95.10,96.13,96.61,96.12,96.17,89.31,-1,149,0,149,1864
,1379355276,108,0.049,85.71,92.24,95.10,96.13,96.61,96.12,96.17,89.31,-1,149,4,145,17472   <<< Write Latency
,1379355276,112,0.049,84.25,90.40,93.20,94.21,94.68,94.20,94.25,87.56,0,148,3,146,1864
,1379355276,116,0.049,84.03,90.35,93.26,94.38,94.99,94.62,94.25,87.56,1,147,3,146,2608
,1379355276,120,0.049,84.03,90.35,93.26,94.38,94.99,94.62,94.85,88.27,2,146,3,146,1936
,1379355276,123,0.049,85.45,92.00,94.91,96.02,96.59,96.14,96.33,89.51,3,145,2,147,1848
,1379355276,127,0.049,85.45,92.00,94.91,96.02,96.59,96.14,96.33,89.51,2,146,2,147,2232
,1379355276,131,0.049,83.88,90.16,93.01,94.10,94.66,94.22,94.40,87.84,2,146,1,148,1860
,1379355276,135,0.049,83.88,90.16,93.01,94.10,94.66,94.22,94.40,87.84,1,147,1,148,1936
,1379355276,140,0.049,84.64,91.32,94.33,95.55,96.24,95.88,96.15,89.40,0,148,1,148,3132
,1379355276,144,0.050,86.05,92.79,95.81,97.00,97.66,97.24,96.15,89.40,0,148,0,149,1804
,1379355276,148,0.050,86.05,92.79,95.81,97.00,97.66,97.24,97.45,90.49,-1,149,0,149,1764

Hey, thanks for the reply. I will spend some time later this week checking it out.

But let me see if I understood.. Now I understand the NilRTOS is sort of a non-preemtive Operating System, that in simple words, is a framework for the code to run such that you can easier divide the work into the different tasks. In general, thats a good idea for sure. i worked on one back in 2001 for a PIC microcontroller.

The only thing I didnt quite understood is how would this necessarely help on my issue. And sorry, for sure I will read your recommended links. But even with a better structure, what exactly in that RTOS helps me deal with the Serial interrupt whilre writing to SD? If I find that "detail", in theory I could implement even without the RTOS right?

Thanks again!!

Now I understand the NilRTOS is sort of a non-preemtive Operating System

It is a preemptive RTOS..

But even with a better structure, what exactly in that RTOS helps me deal with the Serial interrupt whilre writing to SD

Based on your problem description above - I do not understand why you think you have problems with Serial interrupt. Serial is a subsystem which includes interrupt processing and buffering. You do not need to touch it.

You are missing your serial data you want sniff because your approach is not optimal. You need "something" which is able to work "in parallel":

a) receiving the data via serial, and

b) writing the data to the sdcard, and

c) to buffer the data somehow because of sporadic sdcard's write latencies, and

d) the above a), b), c) done in parallel.

No busy-wait operation (waiting while "something else" happens), asynchronous operation with a buffering mechanism to avoid latencies.

200bytes from serial to catch and write to an SDcard is easily doable with NilRtos and 2-3 simple tasks I think. You have to provide some additional info on your problem however - ie. how often the 200bytes data package needs to be written to the SDcard. Based on my experience with Nilrtos, you can easily write a continual data stream (ie. text data) up to 20kB/sec to the sdcard without loosing a single byte.

The assumption here is you have enough RAM for creating a sufficient fifo buffer to compensate the sporadic sdcard's outage. For 20kB/sec (eqv. ~230kbaud serial speed) and a worst case sdcard's write latency of 300ms you need ~7kBytes large fifo buffer..

Hi Borgeje, Were you able to run with the intended speed using Nitros? -Pawan

You could hack your copy of the Arduino runtime to use a larger read buffer in HardwareSerial class...

Hey Pito and MarkT,

I was wondering how RTOS, an operating system is to be implemented with the Arduino by just using libraries. I don’t quite get the concept here. Can you explain in simple terms how is RTOS can be used with Arduino? Sorry, I am new to Arduino but I am trying to learn more! I have a similar datalogging project that might need to use RTOS.