next topic »

[on:]

Serial Data Logger to SD card

I would like to start by thank everyone for their contributions; past, present, and future;

For the past several weeks I have been working on a project  that allows me to receive ASCII or binary data from an external device and save that data directly to an SD card.

I have read a great deal about the subject and tried every library and sketch that I could find.

The external device is a specialized GPS box that provides a steady stream of data (much more accurate and in-depth than the GPS shields can provide).  

This device normally connects to a PC via a null-modem cable and spits out the data asynchronously into a hyperterminal or proprietary software.

The com port on the device is configured like so...

RS232
9600 Baud
No Parity
8 Bits
1 Stop Bit
No Hand Shaking
Echo Off

When powered up it automatically starts the flow of data autonomously.

Here is a snippet of the ASCII data as it is received in hyperterminal...

Code:

$PVAA,505,279936.20,-2546917.502,-4611658.241,3583504.079,0.033,0.007,0.001,0.22.00,0.000,0.000,0.000,0.000,0.000,1,5*3B                              
$AT
$PVAA
4,-0.142,0.036,1,1,1*29000,0.000,0.000,0.000,0
$PVAA,505,279936.40,-2546917.491,-4611658.337,3583504.053,-0.073,-0.080,0.043,-0            
$ATTA,0
ªDXÍÌÌl'TXÁ43333³0@=$PVAA,0,0.00,100.000,0.000,0.000,0
.532,-0.527,0.316,1,1,1*0A,0,0*37                  
$PVAA,505,279936.60,-2546917.462,-4611658.314,3583504.012,-0.061,-0.036,0.053,-0.000,0.000,
$SPHA,0,0.00,0.000,0.000,0.000,1*3B0,0,0*37

     


I have a good binary to ASCII file converter for this type of data so recording it in either fashion is fine.



I have had the most luck with the FileLogger library

http :// code.google.com/p/arduino-filelogger/

for recording the data I receive.

My current dilemma is receiving the data in arduino from the external device and neatly handing it over to the FileLogger routine. I am losing bits and have a significant lag.

Here is my code. I know it is dirty and messed up, but I have chased my tail for a good two days now and am desperate.

Code:

#include <ctype.h>

#define bit9600Delay 84  
#define halfBit9600Delay 42
#define bit4800Delay 188
#define halfBit4800Delay 94

byte rx = 6;


#include "FileLogger.h"

// variable used when reading from serial
byte inSerByte = 0;

#define MESSAGE "1"
unsigned long length = sizeof(MESSAGE)-1;
//byte buffer[] = MESSAGE;


void setup() {
   Serial.begin(9600);
  pinMode(rx,INPUT);


}

void loop()
{

  
    byte val = 0;
  while (digitalRead(rx));
  //wait for start bit
  if (digitalRead(rx) == LOW) {
   delayMicroseconds(halfBit9600Delay);
    for (int offset = 0; offset < 8; offset++) {
     delayMicroseconds(bit9600Delay);
     val |= digitalRead(rx) << offset;
    
    }
    //wait for stop bit + extra
    delayMicroseconds(bit9600Delay);
    delayMicroseconds(bit9600Delay);
  }
  
byte buffer[] = {val,val};

 FileLogger::append("gps.txt", buffer, 8);
 Serial.print(digitalRead(rx));

  
  
}

I am very thankful for any help provided.

[Reply #1 on:]

I'm just starting to work with the FileLogger library (and have my own problems  :) so I may not be much help.

However, I might suggest printing your buffer and seeing how it compares with what's on the card. Then you know if the problems in the read or append. I also note you are always writing a fixed 8 bytes at a time. I guess you are doing that because it is a stream of data, but if you could build a longer string it might reduce the overhead. I too have noticed a lag, but I'm not sure if it's in the string work leading up to the write. The append is advertised as being pretty fast.

So far FileLogger appears to be a nice little clean library, and I have high hopes.

[Reply #2 on:]

Give NewSoftSerial a try.  Using an interrupt driven serial may make life simple

[Reply #3 on:]

Hi Gurt,

I can give you some clues. The code will not work, as you declare a two bytes buffer but try to write eight bytes. Intead try this:

// remove this line... byte buffer = { val,val};

 FileLogger::append("gps.txt", &val, 1);

That's it, it should work. If you want to buffer data before writing it (for instance, wait till 32 bytes are read before writting) let me know and I'll give you the code.

Regards,

Eduardo García.

[Reply #4 on:]

Eduardo,
I didn't want to take this off topic, so I posted a problem I have at
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1241105607/0#0 just in case you might have a clue.

[Reply #5 on:]

Okay, thank you very much for the pointers. I  made some changes to my code (including a record-enable function so that I can stop the recording and keep the file intact). I also implemented newsoftserial as suggested.

Here is what I now have.

Code:

/*
Serial to SD card DATA logger
*/

#include <ctype.h>

#include <NewSoftSerial.h>

NewSoftSerial mySerial(6, 7); //pin 6 RX from device pin 7 TX to device
byte writing = 2;  //LED writing indicator
byte recordenable = 5; //record enable pin


#include "FileLogger.h"

// variable used when reading from serial
byte inSerByte = 0;


void setup() {
  Serial.begin(115200);

  pinMode(recordenable, INPUT);
  pinMode(writing, OUTPUT);
  
  Serial.println("Online!");

  // set the data rate for the NewSoftSerial port
  mySerial.begin(9600);
  mySerial.println("Begin");

}


void loop()
{  
  
  byte val = 0;
  
  while(digitalRead(recordenable) == HIGH) { //check for record enable signal
    
    digitalWrite(writing,HIGH); //indicate that recording is active
  
 
  if (mySerial.available()) {
    val = (char)mySerial.read(),BYTE;
    
    int long length = sizeof(val) - 1;
    
     //FileLogger::append("GPS.txt", &val, 1);
    
    Serial.print(val);  
}
  
  }
  
  while(digitalRead(recordenable) == LOW) { //check for record enable signal
   digitalWrite(writing,LOW); //indicate that recording is inactive
  }
  
}

Right now, when I monitor my PC PC serial port in Arduino this is what I see ...

Code:

Online!

[]]õ?ù;û7û÷vn}ù???ùû÷öv,ö6ìöövì?´öövì6ìöövì6ìööæÙvV6ìöövlÖ6ìööùWW}§?§?£??§{??ss§?£???§??£???§?£???§?£???§?£???§???£???§??£???§?£???§?«?{åë«ïݽýÿÿÿOÿÿÿ
ûÿÿÿÿÿÿ?|É}A/ÿ??2}#?}Î,?ü$Ù*7?rþ?|Ý1 àýûÿÿÇÇߢéñÿ?Õ·"?õßÇ?rÿÿÿÿ«ïÝñ?ÿÿÿGûÿÿ
ýÿÿÿÿÿÿG?É}ÿÿÿÿéÿÿÿïÿÿÿ÷?ÿÿÿï[Y§aÒþüÿÿ?Ãÿs9Yá«Ä?©ZGá÷*òüU(½iö?1}ÿÿÿÿñÿÿÿÓ?ÿÿÿsÆB?¯þþÿÿÏùJ§ÊN??ãÿÿÿÿÿÿÿÿ÷/bÂõ?]é?ÿÿÿåÿÿÿ·?ÿÿÿY??ONþüÿÿ¯¶êy«'3Â,èXv7B×öçå¹jV|t?êýÿÿÿÙÿÿÿ??ÿÿÿéîÚÜþþÿÿ7éfâý½è
¬éëÞô,½£1õÿÇÕ?&[Ñ}ÿÿÿÿÍÿÿÿw?ÿÿÿ©?wvþþÿÿ/M?þþ}?ê©?¯bBHGÁíTÎ?ÜõÙS&ò*ôýÿÿÿÝÿÿÿW?ÿÿþ*Ä?\ôXùýÿÿ5éúþÝ}?¹kTæÿÿÿÿ_ÏiyKÉ}ÿÿÿÿéÿÿÿ7?ÿÿÿõ¢^©?þþÿÿÇO?ù      £:??R¤YQs>¯ÿÿ?      2?»Ã'}ÿÿÿÿÂÿÿÿ?ÿÿÿ?ç®"î«ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿíÿÿÿÇÿÿÿ÷ÿÿÿã+      *>öþüÿÿ/imé?W)?[?ëQs>ÿ·þdL;¨?}ÿÿÿÿëÿÿÿ×ÿÿÿ?»¿Ç©?þþÿÿg-!ÏGö"Tî??????zɽÛ+C?}ÿÿÿÿÿÿÿ½¿ýÿþñ?RÔþþÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿþÿÿÿÿÿÿÿÿÿÿÿíÿÿÿ

When using HyperTerminal and viewing rhe output of the GPS device, this is what I see...

Code:

$PVAA,505,438277.05,-2546918.663,-4611650.436,3583500.395,0.065,0.037,0.194,0.151,0.067,0.332,1,1,1*03
$PVAA,505,438277.15,-2546918.628,-4611650.414,3583500.421,0.021,-0.086,0.000,0.054,-0.546,0.380,1,1,1*08
$PVAA,505,438277.20,-2546918.624,-4611650.407,3583500.406,-0.068,0.027,0.048,-0.152,0.451,0.478,1,1,1*0F
$PVAA,505,438277.30,-2546918.606,-4611650.358,3583500.383,0.043,0.029,0.031,0.119,0.208,0.144,1,1,1*0F
$PVAA,505,438277.35,-2546918.616,-4611650.359,3583500.362,-0.146,-0.141,-0.069,0.163,0.251,-0.766,1,1,1*04
$PVAA,505,438277.45,-2546918.608,-4611650.358,3583500.290,0.123,0.187,-0.062,0.381,0.458,0.097,1,1,1*26
$PVAA,505,438277.50,-2546918.624,-4611650.417,3583500.312,0.023,0.073,-0.069,-0.961,-0.807,-0.055,1,1,1*0A
$PVAA,505,438277.60,-2546918.523,-4611650.321,3583500.279,-0.097,-0.076,-0.023,-0.851,-0.658,-0.170,1,1,1*07
$PVAA,505,438277.65,-2546918.520,-4611650.313,3583500.280,0.091,0.145,0.059,0.222,0.384,0.208,1,1,1*0A
$PVAA,505,438277.75,-2546918.552,-4611650.322,3583500.244,0.026,-0.024,-0.011,0.204,-0.048,0.131,1,1,1*21
$PVAA,505,438277.80,-2546918.554,-4611650.337,3583500.341,-0.062,-0.071,0.069,-0.103,0.438,-0.117,1,1,1*0C
$PVAA,505,438277.85,-2546918.560,-4611650.340,3583500.342,-0.052,0.003,-0.042,0.101,0.326,0.101,1,1,1*0F
$PVAA,505,438277.95,-2546918.706,-4611650.493,3583500.333,-0.118,-0.079,-0.063,-1.060,-0.733,-0.588,1,1,1*00
$ATTA,0,0.00,B00FF,0.000,30.000,0.000,0.000,0.000,245.000,61.000,0.000,0*5B
$PVAA,505,438278.20,-2546918.756,-4611650.742,3583500.662,-0.044,-0.092,0.081,-0.392,-0.566,0.443,1,1,1*00
$PVAA,505,438278.30,-2546918.763,-4611650.708,3583500.635,0.007,0.029,0.031,-0.164,0.318,0.395,1,1,1*22
$PVAA,505,438278.40,-2546918.586,-4611650.484,3583500.595,0.070,0.087,-0.062,0.361,0.320,-0.418,1,1,1*03
$PVAA,505,438278.50,-2546918.572,-4611650.477,3583500.596,0.046,0.179,-0.136,0.040,0.802,-0.750,1,1,1*07
$PVAA,505,438278.60,-2546918.463,-4611650.357,3583500.495,-0.002,0.026,0.011,0.063,0.762,-0.339,1,1,1*0E
$PVAA,505,438278.75,-2546918.684,-4611650.524,3583500.434,0.020,0.072,-0.074,0.332,0.577,-0.466,1,1,1*06
$PVAA,505,438278.85,-2546918.671,-4611650.551,3583500.388,-0.095,-0.051,-0.012,-0.271,-0.326,0.330,1,1,1*23
$PVAA,505,438278.95,-2546918.606,-4611650.462,3583500.368,0.047,0.041,-0.023,0.324,0.420,-0.244,1,1,1*0E
$PVAA,505,438279.10,-2546918.482,-4611650.261,3583500.311,0.011,0.047,-0.046,-0.033,0.392,-0.405,1,1,1*24
$PVAA,505,438279.25,-2546918.399,-4611650.179,3583500.515,-0.052,-0.102,-0.055,-0.777,-0.763,0.137,1,1,1*2B
$PVAA,505,438279.35,-2546918.379,-4611650.180,3583500.509,0.050,0.054,0.091,-0.146,-0.429,0.918,1,1,1*06
$PVAA,505,438279.55,-2546918.430,-4611650.208,3583500.573,0.009,0.021,-0.025,0.158,-0.074,-0.158,1,1,1*27
$PVAA,505,438279.60,-2546918.422,-4611650.292,3583500.692,0.011,-0.067,0.085,0.018,-0.563,0.862,1,1,1*07
$PVAA,505,438279.70,-2546918.434,-4611650.321,3583500.704,-0.044,-0.090,0.087,-0.019,-0.234,0.400,1,1,1*00
$PVAA,505,438279.85,-2546918.519,-4611650.491,3583500.813,-0.001,-0.094,0.044,-0.677,-1.522,0.454,1,1,1*05
$PVAA,505,438279.95,-2546918.491,-4611650.466,3583500.811,-0.022,-0.000,-0.117,0.349,0.712,-0.652,1,1,1*08
$SPHA,505,438280.05,0.078,248.150,0.045,0*0E

Any suggestion why the data received in Arduino appears to be garbage?

I tried using..
val = (char)mySerial.read();
val = (char)mySerial.read(),BYTE;
val = (char)mySerial.read(),DEC;
val = (char)mySerial.read(),BIN;
val = (char)mySerial.read(),OCT;
val = (char)mySerial.read(),HEX;

Once I can get the data under control in the serial interface, I will work on the actual recording. (I tried recording it and it was dropping bits like crazy)

When I did enable this line...
FileLogger::append("GPS.txt", &val, 1);

I got this on the card...

Code:

¡4–^§ýõy9{´“ ïu)[Qs ÿÝÿÿÿéÿÿÿÇÿÿÿëÿÿÿ·}WW}§Ÿ§Ÿ£Ÿ

Obviously I need to use a buffer. Could use help fixing that.



Thank you again!

Garrett

[Reply #6 on:]

OK...

I have made some progress. Using the code below, I can see my data coming in to the arduino properly.

Code:

 /*
Serial Data Logger
 */

#include "FileLogger.h"

 #include <string.h>
 #include <ctype.h>

 int ledPin = 13;                  // LED test pin
 byte writing = 2;                 //LED writing indicator
 byte recordenable = 5;            //record enable pin
 int byteGPS=-1;
 char linea[300] = "";

 void setup() {
   pinMode(ledPin, OUTPUT);       // Initialize LED pin
   pinMode(recordenable, INPUT);
   pinMode(writing, OUTPUT);
   Serial.begin(9600);
   for (int i=0;i<300;i++){       // Initialize a buffer for received data
     linea[i]=' ';
   }  
 }

 void loop() {
   digitalWrite(ledPin, HIGH);
  
   while(digitalRead(recordenable) == HIGH) { //check for record enable signal
    digitalWrite(writing,HIGH); //indicate that recording is active
    
   byteGPS=Serial.read();         // Read a byte of the serial port
   if (byteGPS == -1) {           // See if the port is empty yet
     delay(100);
   } else {
    
     unsigned long lengthofGPS = sizeof(byteGPS) -1 ;
    
     //FileLogger::append("GPS.txt", (byte)byteGPS, lengthofGPS);   // HELP ME PLEASE
    
     Serial.print(byteGPS,BYTE);
   }
  
   }
 }

byteGPS, BYTE is the data as I need it recorded to the text file.

I have played with FileLogger and cannot properly format the FileLogger::append statement to get it to write byteGPS to the file.

I am guessing that this is a simple issue of semantics, but I'm lost. :'(

uFat will not work for me since the recorded data file might get huge (200 megs possibly)

Thank you again for your help,
Garrett

[Reply #7 on:]

awesome i was going to try something like this

my gps spits out TTL serial data via a cable at 38400 baud, it also does bluetooth at 4800-19200

i have a couple of spare SD cards waiting for use too :3

[Reply #8 on:]

try replacing this

//FileLogger::append("GPS.txt", (byte)byteGPS, lengthofGPS);

with this:

byte byteToWrite = (byte)byteGPS;
FileLogger::append("GPS.txt", &byteToWrite, 1);

[Reply #9 on:]

Hi Garrett,

All I can offer in terms of advice is:

* Get things working one component at a time. Serial reception, then storage. Attempting to troubleshoot 2 systems at once is going to give you a big headache!

* Concentrate on the types of data you're receiving, and working out the simplest conversion. It's very important that you understand the data types and how the compiler manages the conversion between them.

Here's some code that buffers data for you. You'll need to experiment with the buffer size to achieve a steady data flow. You will need to write as much data as you can without missing the incoming data.

If you can afford 512 bytes, then do so as this is the unit size of data transfer for mmc/sd. I assume the library has a special case for this block size - if it doesn't then consider changing to one that does. I can't help with specifics as I'm not familiar with this library.

Once this code works for you, then add the record-enable code around it.

Code:

int bufferIdx = 0;
byte buffer[512];

void loop(void)
{
  while(Serial.available())
  {
    byte val = (byte)Serial.read();
    if (bufferIdx == 512)
    {
      FileLogger::append("GPS.txt", buffer, 512);
      bufferIdx = 0;
    }
    buffer[bufferIdx] = val;
    ++bufferIdx;
  }
}

Hope this helps,

Charlie


PS: Why do you think uFat can't write to large files? uFat can address every sector on a card, as long as the file being written to is pre-allocated. It may not be the ideal solution for all tasks, but when you need tiny code it's unbeatable IMHO

**edited to fix code**

[Reply #10 on:]

I think

Code:

buffer[bufferIdx] = val;

should be

Code:

buffer[bufferIdx++] = val;

?

[Reply #11 on:]

You're right. I've updated the original post.

Thanks

[Reply #12 on:]

SUCCESS!!

Thank you all very much, especially edugarcia and sirmorris. Your contribution made this success possible.

I have been programming for years and designing electronics for years but not both combined. Arduino got me into that. This project might seem trivial to most, but there are many pitfalls.


Here is my finished code. With it I can record binary straight from a serial device using softserial and a MAX232 right to an SD card.
Very beneficial to data collectors!

Code:

/*
Serial Data Logger
 */

#include "FileLogger.h"

 #include <string.h>
 #include <ctype.h>

 #include <NewSoftSerial.h>

NewSoftSerial GPS(6, 7); //pin 6 RX from device pin 7 TX to device

 int ledPin = 13;                  // LED test pin
 byte writing = 2;                 //LED writing indicator
 byte recordenable = 5;            //record enable pin
 int byteGPS=-1;
 int bufferIdx = 0;
 byte buffer[512];


 void setup() {
   pinMode(ledPin, OUTPUT);       // Initialize LED pin
   pinMode(recordenable, INPUT);
   pinMode(writing, OUTPUT);
   Serial.begin(9600);
   GPS.begin(9600);

 }

 void loop() {
  
   while(digitalRead(recordenable) == HIGH) { //check for record enable signal
    digitalWrite(writing,HIGH); //indicate that recording is active
    
     while(GPS.available())
  {
    byte val = (byte)GPS.read();
    if (bufferIdx == 512)
    {
      FileLogger::append("GPS.gps", buffer, 512);
      bufferIdx = 0;
    }
    buffer[bufferIdx] = val;
    ++bufferIdx;
  }

  
   }
  
   while(digitalRead(recordenable) == LOW) { //check for record enable signal
   digitalWrite(writing,LOW); //indicate that recording is inactive
  
   //IDLE
  
  }
 }