Handling Multi-line serial data

Projects Programming
1

Hello all,

This is a carry over from another thread i started but the conversation is starting to deviate.

I have an arduino uno communicating serially with an RF modem. At this point I'm basically issuing it some AT commands and looking at the response. The responses come on multiple lines with terminators.

Code:

#include <SoftwareSerial.h>

SoftwareSerial monSerial(10, 11); //Rx, Tx

void setup() {
  Serial.begin(38400);
  Serial.setTimeout(500);
  monSerial.begin(38400);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
}

void loop() {  
  readSer();
  checkConn();
  Serial.print("AT+CNUM?\r\n");
  delay(100);
  readSer();
  delay(5000);
}

void readSer() {
  
  const int buffSize = 50; //Set buffer size
  char sData[buffSize]; //Array for storing serial data
  char sChar; //Character read from serial
  int len =0; //length of what's in buffer

  while(Serial.available() > 0) //only read if data
  {
    sChar = Serial.read(); //Read it
    if ((sChar == '\r') || (sChar == '\n')) {
      //EOL
      if (len > 0) {
        sParse(sData);
      }
      len = 0;
    }
    else {
      if (len < buffSize) {
        sData[len++] = sChar; //append recieved char to array
        sData[len] = '\0'; // append null terminator to end
      }
    }
  }
}

void sParse(char *msg) {
  const char * rsp = "+CNUM: 0,";
  char * p;
  int ID;
  p = strstr(msg, rsp);
  if (p) {
    char *ISSI;
    ISSI = strrchr(p, ',');
    ISSI++;
    ID = strtoul(ISSI, NULL, 16);
    monSerial.println(ID);
  }
  rsp = "+CTSDSR:";
  p = strstr(msg,rsp);
  if (p) {
    monSerial.println("Status Received");

readSer() feeds the incoming data into a char array and once it his a or it will send it to sParse(). Eventually sParse will be evaluating the data on multiple criteria.

As it is now I'm just looping through making sure the modem is connected. Occasionally the modem will receive a message from another modem which is why i call readSer() at the beggining of the loop. in sParse(), I'm looking for a line that has +CTSDSR

The actual message from the modem looks like this

+CTSDSR:13,0,1010,0,1052,16<cr><lf>
814D<cr><lf>

My problem is that the data of interest (814D) happens on the next line after what's caught by my if statement. But because my readSer() function is stripping out the cr/lf and passing the data one line at a time i can't get it. The message can change

I tried calling readSer() from inside the if statement to get the next line, it compiled but nothing happened so i can only assume that's not proper... Since I know from the message that what I want is on the next line so could do Serial.read, but then i have to make another buffer to read the data which makes the readSer() function redundant...

I'm sure this is childs play for some but at this point my brain is melting trying to figure out the best way to approach this.

2

The answer is "state machine" but that doesn't help you get one.

Once you've recognised the +CTSDSR line, then you need to remember that. Then when the next line is parsed, you look in your memory to see if it's a special line. If it is the special line, do the special processing and then reset the memory so that you're looking for +CTSDSR again.

The phrase "state machine" means the machine remembers what state it's in. Are we waiting-for-CTSDSR or are we got-CTS-now-waiting-for-the-next? Those are two "states".

3

Your method of reading the serial data assumes that there is a complete packet in the buffer, and nothing more than a complete packet. There is rarely going to be true.

4

kmarts:
This is a carry over from another thread i started but the conversation is starting to deviate.

Is this the other Thread?

Including a link to it will help people to avoid repeating themselves or someone else.

...R

5

The second example in Serial Input Basics will capture each line up to the LF character. As you get each line you can test it as suggested by @MorganS

...R

6

Robin2:
Is this the other Thread?

Including a link to it will help people to avoid repeating themselves or someone else.

...R

Yes, sorry about that.

Robin2:
The second example in Serial Input Basics will capture each line up to the LF character. As you get each line you can test it as suggested by @MorganS

...R

Ok so going back to the last thread. I broke up the reading a parsing into two functions. If I'm to test the line i would need to do it in the readSer() function correct?

7

kmarts:
If I'm to test the line i would need to do it in the readSer() function correct?

I would not.

In my mind gathering data is a separate activity from interpreting the data.

...R

8

Robin2:
The second example in Serial Input Basics will capture each line up to the LF character. As you get each line you can test it as suggested by @MorganS

...R

Hi Robin,

I've kind of been using this writeup (which is great btw) as my guide thus far.

I changed my code from the original so the serial data array is in the global scope and am trying to use a Boolean 'newData' variable to control reading the next line as per your example(s) but now I'm experiencing some odd behavior.

As a simple test this is what I've done:

#include <SoftwareSerial.h>

SoftwareSerial monSerial(10, 11); //Rx, Tx

//Variables for serial data
const byte buffSize = 50; //Set buffer size
char sData[buffSize]; //Array for storing serial data
char tempChars[buffSize]; //temp array for parsing
boolean newData = false;

boolean sConn = false;

void setup() {
  Serial.begin(38400);
  monSerial.begin(38400);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
  delay(2000);
}

void loop() {  
  //checkConn();
  if (newData == false) {
    Serial.print("AT+CNUM?\r\n");
  }
  delay(100);
  readSer();
  if (newData == true) {
    monSerial.print(sData);
    newData == false;
  }
  delay(2000);
}

void readSer() {

  char sChar; //Character read from serial
  static byte len = 0; //length of what's in buffer
  
  while(Serial.available() > 0 && newData == false) {//only read if data
    
    sChar = Serial.read(); //Read first byte    

    if ((sChar != '\r') || (sChar != '\n')) {
        sData[len++] = sChar; //append recieved char to array
        sData[len] = '\0'; // append null terminator to end
        //monSerial.print(sData);
    }
    else {
      if (len > 0) {
        newData == true;
        len = 0;
      }
    }
  }
}

So I'm printing the output of sData to a software serial port that i can monitor to see what's going on. AT+CNUM? is only written once, after that nothing happens. So i started printing out the status of newData to the serial port thinking that for whatever reason it wasn't beeing changed, see the attached screen shot.

I can see the program is looping, but the AT command is only written once (there's more in the serial monitor window because the modem wasn't connected for the first few writes).

What am i missing here?

terminal.png
terminal.png1210×913 41.7 KB

9

Image from Reply #7 so we don't have to download it. See this Image Guide

d459c79abf6c22b4340d5e48ae1174cef8d21c42.png
d459c79abf6c22b4340d5e48ae1174cef8d21c42.png1210×913 41.7 KB

...R

10

Please don't post pictures of text as they are unreadable. Just copy and paste the text.

...R

11

Take the delay()s out of your loop(). The function readSer() should be called as often as possible.

If you need to slow the rate of printing to the Serial Monitor see how millis() is used to manage timing without blocking in Several things at a time.

It is much easier to help if you just use the functions in my tutorial without any changes. Put whatever special code you need into other functions.

...R

12

Robin2:
Please don't post pictures of text as they are unreadable. Just copy and paste the text.

...R

Ok, what you see in the terminal is that the status of newData is always false (0) but the arduino stopped writing the AT command to the modem as if it's true and monSerial.print(sData) yeilds no output.

I'm confused

Robin2:
It is much easier to help if you just use the functions in my tutorial without any changes. Put whatever special code you need into other functions.

...R

ok, I'm kind of a hands on learner so was trying to understand what's going on and experience the common gotchas of the language. I understand it can be taxing to regularly try and help someone with a serious lack of experience. I will play with your code and let you know.

Thanks

13

Brain Fart

    newData == false;

This has been a problem and i didn't even realize it... Was trying to set newData, should have been single =.

D'oh

Edit:

Second problem I've found is that having multiple conditions for the if statement:

if ((sChar != '\r') || (sChar != '\n'))

This is likely my syntax...

If i test for only one condition it works. This is also true for the example two code from your tutorial. within the recvWithEndMarker() function, if you create a second end marker variable = '\r' and do the same test, neither returns true. However if i change the above to:

f (!(sChar == '\r' || sChar == '\n'))

it always evaluates as true. Time to start more reading[/code]

14

Time to start more reading

  if(sChar != '\n' && sChar != '\r')
  {
     // Not a carriage return or a line feed
  }
15

kmarts:
Second problem I've found is that having multiple conditions for the if statement:

if ((sChar != '\r') || (sChar != '\n'))

Why do you need the double test.

Just use one of them as the end-marker.

And post the latest version of your program. I can't see your PC screen from here.

...R

16

PaulS:

  if(sChar != '\n' && sChar != '\r')

{
    // Not a carriage return or a line feed
  }

Thank you sir, I had managed to figure it out my illogical thinking not too long after posting... You stare at something too long and you get tunnel vision.

Robin2:
Why do you need the double test.

Just use one of them as the end-marker.

The reason for the double test is because my lines are terminated cr+lf and many of the responses from the modem are prefixed with cr+lf as well. When i was trying to do it just by cr, i was ending up with lf's in my char array and odd things were happening with the parsing of data. I know you're probably laughing at my amateur work here but it's just easier for me to get rid of them.

Robin2:
And post the latest version of your program. I can't see your PC screen from here.

...R

I didn't include my parser function because I'm still working on it. But this is working now as i want it, putting out the serial stream line by line.

#include <SoftwareSerial.h>

SoftwareSerial monSerial(10, 11); //Rx, Tx

//Variables for serial data
const byte buffSize = 50; //Set buffer size
char sData[buffSize]; //Array for storing serial data
char tempChars[buffSize]; //temp array for parsing
boolean newData = false;

boolean sConn = false;

void setup() {
  Serial.begin(38400);
  monSerial.begin(38400);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
  delay(2000);
}

void loop() {  
  if (newData == false) {
    Serial.print("AT+CNUM?\r\n");
  }
  delay(200);
  readSer();
  while (newData == true) {
    sParse(sData);
  }
  delay(2000);
}

void readSer() {

    char sChar; //Character read from serial
    static byte ndx = 0; //length of what's in buffer
    char endMarker1 = '\n';
    char endMarker2 = '\r';
        
    while (Serial.available() > 0 && newData == false) {
        sChar = Serial.read();
        if (sChar != endMarker1 && sChar != endMarker2) {
          if (ndx < buffSize) {
            sData[ndx++] = sChar;
            sData[ndx] = '\0';
          }
                       
        }
        else {
            if (ndx > 0) {
              newData = true;
            }
            ndx = 0;
        }
    }
 }

As for the delays. The two second delay on loop() is just for me to process what's going on... What i upload seldom works as planned the first time.

As for the other short delays. I'm finding without them that I'm not always getting the complete output and it also seems that the faster the arduino goes the less stable/reliable my modem becomes.

17

kmarts:
The reason for the double test is because my lines are terminated cr+lf and many of the responses from the modem are prefixed with cr+lf as well. When i was trying to do it just by cr, i was ending up with lf's in my char array and odd things were happening with the parsing of data.

.....

As for the other short delays. I'm finding without them that I'm not always getting the complete output and it also seems that the faster the arduino goes the less stable/reliable my modem becomes.

I wonder if these two issues are related.

I would just use a single end-marker and, if necessary, remove the other unwanted character from the received data.

if there is a CR/LF sequence at the end of one message and another at the start of the next message then using just (say) LF as the end-marker will result in some short messages with just a CR as the content. That is easily ignored.

My guess is that your delay()s are preventing it working like that.

...R