Go Down

Topic: Real time serial (< 57600 bps) data logging to SD card (Read 341 times) previous topic - next topic

Helium328PU

Hi,
I am trying to log serial data in real time from another Arduino (Mega) to SD card. Slave Arduino is sending 4kb data array to Master. Master device has SD shield. My goal is to do it as quickly as possible (to let Slave continue in its routines). Current serial speed is 57600 bps, experimented with 115200 and it doesnt show any change.
Data format looks like this:
Code: [Select]

* //start of stream
125 //byte 1, NL
...
255 //byte 4096, NL
# //end of stream


Problem is that it looks like that even SDfat library is not (maybe???) fast enough for that. I use this code with 64 byte buffer (there is also commented part where I wrote to file byte by byte).

Code: [Select]

while(1){
  if (Serial.available() > 0) {
    // read the incoming byte:
    //time = micros();
    incomingByte = Serial.read();
    buffer+=incomingByte;
    //Serial.print(incomingByte);
    // Write data to CSV record.
    //file.print(incomingByte);
    if(buffer.length()==64){
      file.print(buffer);
      buffer="";
      writtenBytes=writtenBytes+64;
      }
    //writtenBytes++;
    //time2 = micros();
    //Serial.println(time2-time);
    //delayMicroseconds(100);
    if (incomingByte=='#'){
    //Serial.flush();
    Serial.println(writtenBytes);
    //Serial.println(fileName);
    writtenBytes=0;
    Serial.println("CLOSED! ");
    file.close();
    break;
    }
  }
  }


Output indicates that approximately only 6800 - 7000 bytes are written to SD (should be exactly 8192 bytes all the time).
Its even more strange that when measuring with micros() function it says that writing single byte to file lasts about 48 to 56 uS (max). At 57600 bps one frame transmission takes 156.25us so there should be almost 300% time reserve. I even tried to define larger serial buffer but it seems to have no effect.

Could anyone suggest what else might be the problem? Or what could be solution of solving this? Maybe I have to use Arduino Due (cant find info if it can use faster SPI access)?

PaulS

Code: [Select]
    buffer+=incomingByte;
A String is NOT a buffer.

Appending data to a String instance is SLOW.

Writing to the SD card is already buffered. Writing a 64 byte String is NOT making the process faster.
The art of getting good answers lies in asking good questions.

Helium328PU

#2
Apr 24, 2017, 04:21 pm Last Edit: Apr 24, 2017, 05:01 pm by Helium328PU
Code: [Select]
    buffer+=incomingByte;
A String is NOT a buffer.

Appending data to a String instance is SLOW.

Writing to the SD card is already buffered. Writing a 64 byte String is NOT making the process faster.
Your right, its just a bad name I admit. But you got the point how it was meant.

Actually I thought that maybe writing bigger "chunks" could be faster (like on PC sequential write is always faster).  But made a comparison and its still slow. I carefully made different data samples and noticed that changing size of hardware serial buffer makes a slight difference - files are actually about 200 bytes larger when using 256 bytes instead od default 64.  But there is still loss of about 400bytes.

This is original single byte algorythm, i guess that it cant be optimized to be even faster?

Code: [Select]



#include <SPI.h>
#include "SdFat.h"

// SD chip select pin.  Be sure to disable any other SPI devices such as Enet.
const uint8_t chipSelect = SS;

char incomingByte = 0;   // for incoming serial data
int tick=0;
int writtenBytes=0;
char fileName[13];

// Error messages stored in flash.
#define error(msg) sd.errorHalt(F(msg))

// File system object.
SdFat sd;

// Log file.
SdFile file;

void setup() {
  Serial.begin( 57600 );
  Serial.println(F("Starting up"));
 
  // Initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!sd.begin(chipSelect, SPI_FULL_SPEED)) {
    sd.initErrorHalt();
  }
  Serial.print(F("Logging to: "));
  Serial.println(fileName);
}

void loop() {
String buf ="";;
  buf+="SPEC_";
  buf+=tick;
  buf+=".txt";
  buf.toCharArray(fileName, 13);
  if (!file.open(fileName, O_CREAT | O_WRITE | O_EXCL)) {
    error("file.open");
  }

while(1){
  if (Serial.available() > 0) {
    // read the incoming byte:
    incomingByte = Serial.read();
    // Write data to CSV record.
    file.print(incomingByte);
    if (incomingByte=='#'){
    Serial.println(writtenBytes);
    //Serial.println(fileName);
    writtenBytes=0;
    Serial.println("CLOSED! ");
    file.close();
    break;
    }
  }
  }


  // Force data to SD and update the directory entry to avoid data loss.
  if (!file.sync() || file.getWriteError()) {
    error("write error");
    Serial.println("ERROR!");
  }
  //filename="";
  tick++;
  // do something else later
}


So I think thay probably the only solution is to use Due, allocate 4kbytes of RAM to use as internal buffer....

PaulS

Quote
So I think thay probably the only solution is to use Due
Another possible solution is to visit http://snippets-r-us.com for help with your snippets.

The art of getting good answers lies in asking good questions.

Helium328PU

Another possible solution is to visit http://snippets-r-us.com for help with your snippets.


Well if you are perfectionist, I have added all the minimal code in last post. When I help other people I dont usually thing thats necessary to post full code every time.

But still I doubt that it could change anything.

PaulS

I think your code should look something like this:
Code: [Select]
void loop()
{
   if(Serial.available() > 0)
   {
      // Open a file

      while(Serial.available() > 0)
      {
         // Read a character
         // If not #
         //    Write to file (buffer)
      }

      // close the file
   }
}
The art of getting good answers lies in asking good questions.

Helium328PU

#6
Apr 27, 2017, 11:50 am Last Edit: Apr 27, 2017, 12:16 pm by Helium328PU
Hi PaulS,

thanks for code. But I think its very similar to what I have posted. Anyway, tested it also and got similar "performance". It looks like that SDfat library dont have consistent write speeds, or micros() command is not accurate.

Never mind I have tested Arduino Due as receiver, set SPI HALF_SPEED with same code above and it seems that its much more reliable. I got fully 8192 bytes frames as should be. There is still aproximately 5% data loss, but I guess thats because of not so good wiring (I hope).


Go Up