Serial Streaming question

Hi all,

I've posted this in another forum, but it seems the issue I was having is proving to be more of a programming issue than an interface issue.

I am building a standalone gps logger. The logger must record short bursts of full gps sentences (which are not standard nmea gps sentences with a set sentence length). I seem to be having trouble with structuring code such that full sentences are recorded, rather than chars from different sentences at each point of the loop. Currently, My best code either records full sentences that are recorded as gibberish, or records only portions of each gps sentence. I have experimented with trying to do this many different ways. Here is an example that someone else provided

#include <NewSoftSerial.h>
#include <SD.h>
#include <stdlib.h>
NewSoftSerial nss(3,4);
const int CS=8;
static char dtostrfbuffer[40];
String dataString = "";

void setup() {
  pinMode(CS, OUTPUT);
  
  Serial.begin(115200);
  nss.begin(9600);
  Serial.print("Start");
  if(!SD.begin(CS))  {
    Serial.println("Card Failure");
    while(1);
  }

}

void loop()  {
  if (nss.available()) {
    File dataFile =SD.open("LOG.txt", FILE_WRITE);
    if (dataFile)  {
      char c = nss.read();
      dataFile.print(c);
      dataFile.close();
      Serial.print (c);
    }  else  {
      Serial.println("Couldn't open the log file!");
      while(1);
    }
  }
}

Here is some example code that I've written, which does the sbd timing like i mentioned, and comes with a nice counter for the bursts. It seems to have the same problem as the code above

#include <NewSoftSerial.h>
#include <SD.h>
#include <stdlib.h>
NewSoftSerial nss(3,4);
int CS=8;

int j=0;
void setup()  
{
  pinMode(CS, OUTPUT);
  
  Serial.begin(115200);
nss.begin(9600);
  Serial.print("Start");
if(!SD.begin(CS))
{
  Serial.println("Card Failure");
  return;
}
}

void loop()                     // run over and over again
{
 while (j<10000000){j++;
  
  File dataFile =SD.open("LOG.txt", FILE_WRITE);
  if (dataFile)
  {dataFile.print((char)nss.read());
   dataFile.close();
  }
  else
  {
    Serial.println("Couldn't open the log file!");
  }
for (int i=0;i<4000;i++){
  if (nss.available()) {
     
      {
      
        Serial.print((char)nss.read());
        
       

      }
  }
}
delay (7000);
  Serial.println();
  Serial.println();
  Serial.println();
  Serial.print("COUNTER:");
  Serial.print(j);
  Serial.println();
  Serial.println();
  Serial.println();
  
 }
}

Can anyone possibly provide some insight into streaming serial data as it comes in for logging? Or perhaps how to create an array of unknown size that the serial data is to fill, and then print the array to the sd card?

You need some way to tell when you have reached the start of a new sentence. I don't know what the protocol is that you're receiving, but you might be able to recognise a common sequence of characters at the start or end of a message, or perhaps they are separated by linefeeds or similar. As a last resort you might be able to detect a timing gap between the end of one transmission and the start of the next - although working it out directly from the received character stream would be much better.

each new message begins with "$" each message ends with a carriage return

I'm fine with the idea of creating even an arbitrary timing gap, and losing a couple of recorded lines if that's possible. I just don't know how to program it so that I get even a few complete messages for each burst

Just read the stream until you get a $. read subsequent chars into a suitably sized char array until you get CR. Null terminate the array and then write the resulting string to your file.

Opening and closing the file each time a character arrives is going to result in a buffer overrun condition. Open the file when the $ arrives. Read and write every character that arrives until the arrives. When the arrives, close the file.

What you are doing in the second sketch is plain silly. Much of the data read from the GPS is not going to be written to the SD card.

@PaulS and Wildbill, I know ultimately that is what I have to do, but I have no idea how to do it. I understand why I'm getting the errors I'm getting as I mentioned in the earlier post. I just don't know how to do what I need to do, i.e. create a loop that reads data into an array when it encounters a "$" and logs stops and logs when it encounters a carriage return.

I'm looking for direction on how to do something like that, or resources I can use, b/c currently the syntax for it is beyond my current understanding

I've tried this. It prints nothing from the serial port. Unlike the one before,this code doesn't open and close the SD file every time a character is passed in, hopefully preventing an SD timing overlaod. It's intended to open the file, and print b, the character read from the serial port, for a number of iterations (this would output several sentences) then close the file and pause.

#include <NewSoftSerial.h>
#include <SD.h>
int CS=8;
NewSoftSerial nss(3,4);
int i;

void setup()

{
  pinMode(CS,OUTPUT);
  Serial.begin(115200);
  Serial.println("Start");
 nss.begin(9600);
}

void loop()                     // run over and over again
{File dataFile= SD.open("GPSLOG.txt", FILE_WRITE);
{
  for (int i=0;i<5000;i++){
  if (nss.available()>>1) {
    char b = nss.read();
    if (dataFile){
      dataFile.print(b);
  Serial.print(b);
    }
   }
     }
}
dataFile.close();
delay(20000);
}

The approach I'd take is to receive characters from the serial port and bin them until the start of message appeared.

Accumulate subsequent characters into a char array, with a check to make sure that the array didn't overflow, stopping when the end of message appeared.

Then open the file, append the contents of the buffer to the file, and close the file.

Then go back to the start.

  if (nss.available()>>1) {

What is your intention here?

void loop()                     // run over and over again
{File dataFile= SD.open("GPSLOG.txt", FILE_WRITE);
{
  for (int i=0;i<5000;i++){
  if (nss.available()>>1) {
    char b = nss.read();
    if (dataFile){
      dataFile.print(b);
  Serial.print(b);
    }
   }
     }
}
dataFile.close();
delay(20000);
}

Would you look at this mess? Put each { on a new line, or on the end of the line with the statement, but do it CONSISTENTLY.

NOTHING goes on the line after the {. Period.

Use Tools + Auto Format BEFORE you post code. EVERY time.

Get rid of the damned delay()s. LEARN how to not use them.

Try starting with a working example

see
http://www.ladyada.net/make/gpsshield/

Careful about buffer overflow, try going back to the beginning of the buffer when you get a '$'.

This works out the box if you dont care about loosing the odd message or two.
get cygwin and gcc and write some simple code if you are just starting so you dont have to keep round tripping to the arduino.
By that i mean run it on your PC from bash, to try basic coding skills.

... Simon

@simon, The gps config only makes getting regular nmea sentences easier. I'm getting pseudorange messages and other info from my gps as well which makes those examples not very useful for my purposes.

@PaulS. I apologize for the messy code. I've formatted it, and structured the program to read serial into an array of given size and print that array. I have also gotten rid of the delays.

The code for that is below. It starts and then continuously prints "*****". What I was attempting to do was make an array of size 50 and print data to that array until it reads in 50 characters. After it reads 50 characters it is supposed to print the array. If it isn't actually taking the data and putting it into the array, can someone please explain how to get incoming data into the array.

#include <NewSoftSerial.h>
NewSoftSerial nss(3,4);

char message[50];

void setup()  
{

  Serial.begin(115200);
  Serial.println("Start");
  nss.begin(9600);

}


void loop()                    
{


  if (nss.available()) 
  {
    char b = nss.read();
    if (b=message[50])
    {
    Serial.print(message[50]);
    
  }
}

}
    char b = nss.read();
    if (b=message[50])

The second line assigns to b.

You mean, possibly:

    char b = nss.read ();
    if (b == message [50])

Even then, don't you want to print what you read?

void loop()                    
  {
  if (nss.available()) 
    {
    char b = nss.read();
    Serial.print(b);
    }
  }

You mean, possibly:

But not exactly. The upper index on a 50 element array is 49.

Even then, it seems unlikely that the element at index 49 will contain the letter you just read.

Nick, I changed the assignment code you mentioned, however after the "start" check in the code, I just get "********" in the serial monitor.

My end goal is to log sentences of incoming serial data. It is my assumption that in order to log sentences of b, that I need to declare an array, have these serial characters recorded to the array, and then log the array. It is my experience that otherwise I only get smatterings of each sentence as the program loops every time to print a single character.

@PaulS,

by what you just said, do you mean that as it loops, it ends up skipping serial characters?

PaulS:

You mean, possibly:

But not exactly. The upper index on a 50 element array is 49.

Even then, it seems unlikely that the element at index 49 will contain the letter you just read.

Quite right.

Nick, I changed the assignment code you mentioned, however after the "start" check in the code, I just get "********" in the serial monitor.

Even when you printed "b" as I showed in the next post?

It is my assumption that in order to log sentences of b, that I need to declare an array, have these serial characters recorded to the array, and then log the array.

All correct.

char inData[120]; // An array that can hold 120 characters, including the trailing NULL
byte index = 0; // An index into that array that gets incremented each time a character is added

void loop()
{
   while(Serial.available() > 0)
   {
      char inChar = Serial.read();
      if(index < 119)
      {
         inData[index++] = inChar;
         inData[index] = '\0';
      }
   }

   // Do other stuff
}

Now there is a lot more that needs to be done here. You need to determine if a character is a trigger, before adding it to the array. That trigger might be a start of packet marker that would cause you to reset index and put a NULL in position 0. It might be an end of packet marker that would trigger you to use the data in inData, which then represents a complete packet. That use might involve just storing the data, or it might involve parsing it and storing only some of the tokens.

But, it will take care of the I only get smatterings of each sentence as the program loops every time to print a single character" problem.