Locate specific line in serial buffer

Projects Programming
1

I am working on a sensor project where I need to process the results that have been read into a serial buffer. I am looking for the line that contains a +IPD at the beginning and then want to process the remainder of the string as key value pairs. This is what I have so far.

if(wait_for_esp_response(3000, "Unlink")){ 

dbg.print("Contents of buffer inside of getSensorConfig is: ");
dbg.println(buffer);
dbg.println("End of buffer dump................");

if(strncmp(buffer, "+IPD,", 5)==0) {
  // return format: +IPD,ch,len:sensor config data
  dbg.println("We have +IPD!");
  sscanf(buffer+5, "%d,%d", &ch_id, &packet_len);
  if (packet_len > 0) {
    pb = buffer+5;
    while(*pb!='\r\n') pb++;
    pb++;
    dbg.print("Printing contents of pb.....");
    dbg.println(pb);

  }
}

I know that I have a buffer full of of string data as I can println dump the contents of buffer and get the following:

Contents of buffer inside of getSensorConfig is:
AT+CIPSEND=1,74

GET /methods/getsensorconfig?keyId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SEND OK

+IPD,1,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422035449201
OK

OK
Unlink
End of buffer dump................

I thought I could simply strncmp to locate the line beginning with +IPD and and read character by character into pb until \r\n is found.

The problem is that I never seem to be getting into the if block.

Any suggestions on how to do this?

Thanks,

Stephen

2

strncmp() compares whole strings not parts of them.

Have a look at strstr()

3

Hi,

This got me into the if conditional but am I on the string I want? The contents of pb never printed in the println debug statement. Additional println's show ch_id and packet_len equal to 0.

Stephen

4

This got me into the if conditional but am I on the string I want?

What is "This"?
How do we know if you're on the string you want?
We're not psychic.

5

but am I on the string I want?

Here is an off the wall idea. Maybe you could print what strstr() returns.

6

Hi,

I realize you are not psychic. Please read my original post again. I stated the following:

"I am looking for the line that contains a +IPD at the beginning and then want to process the remainder of the string as key value pairs."

Using if(strstr(buffer, "+IPD,")) actually evaluated true so I went into the if statement. So now that I am in the if statement my question is since I have located the line with +IPD am I on this line for the

sscanf(buffer+5, "%d,%d", &ch_id, &packet_len);

statement to execute.

The output I get when I println's for ch_id and packet_len are each 0 now.

So my code now looks like this:

if(wait_for_esp_response(3000, "Unlink")){ 

    dbg.print("Contents of buffer inside of getSensorConfig is: ");
    dbg.println(buffer);
    dbg.println("End of buffer dump................");
    
    if(strstr(buffer, "+IPD,")) {
      // request format: +IPD,ch,len:sensor config data
      dbg.println("We have +IPD!");
      sscanf(buffer+5, "%d,%d", &ch_id, &packet_len);
      dbg.println(ch_id);
      dbg.println(packet_len);
      if (packet_len > 0) {
        pb = buffer+5;
        while(*pb!='\r\n') pb++;  //was :
        pb++;
        dbg.print("Printing contents of pb.....");
        dbg.println(pb);
        //        if (strncmp(pb, "GET / ", 6) == 0) {
        //          dbg.println(F("Content string for configuration"));
        //      }
      }
    }

and my output looks like this:

Contents of buffer inside of getSensorConfig is:
AT+CIPSEND=1,74

GET /methods/v1/getsensorconfig?keyId=c35ba5da9cdbe49a156c251dad6c89b2

SEND OK

+IPD,1,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422039765028
OK

OK
Unlink
End of buffer dump................
We have +IPD!
0
0

I hope this is a bit more clear. Sorry if it is not. Printing strstr() may be a great idea. I will look at it tonight.

Thanks for the help,

Stephen

7

So my code now looks like this:

No, it does not! That snippet won't even compile. Quit wasting our time with snippets. Post ALL of your code, or fix the problem yourself.

8

You could capture the serial character string as a String, then use the Various String functions to parse out the desired data. Is the character string ended with a delimiter or some type of end of data marker?

9

Maybe some of the examples in serial input basics will be useful. They capture all the input into a char array before doing anything else. There is also a parse example.

There are many ways to find things in a char array including brute-force iteration over the array.

In your example maybe you could use the + character as a start marker so that any data before it is discarded. Maybe the D in +IPD could be the start marker if there is no D elsewehere in the data. And you could probably extend my examples to use the whole +IPD as a start marker.

...R

10

Hi Zoomkat\Robyn,

Thank You for the help..... I almost have this working using start and end markers. The idea and tutorial was very helpful. My function looks like this:

String getConfigFromBuffer(char arr[]){    

  monitor.println("Inside getConfigFromBuffer function");
  //  monitor.println("Buffer passed into function:");
  //  monitor.println(buffer);
  //  monitor.println();

  static boolean foundSettingsString = false;
  static byte ndx = 0;  
  char startMarker = '+IPD,';
  char endMarker = '\0';
  char rc;

  for (int i = 0; i < BUFFER_SIZE; i++){

    if (foundSettingsString == true){
      if (buffer[i]!= endMarker){
        configBuffer[ndx] = buffer[i];
        ndx++;        
        if (ndx >= CONFIG_BUFFER_SIZE) {
          ndx = CONFIG_BUFFER_SIZE -1;
        }
      }

      else{
        buffer[i] = '\0';
//        configBuffer[ndx] = '\0';  //terminate the string
        foundSettingsString = false;
        ndx = 0;
        newSettings = true;
      }
    }
    else if (buffer[i] == startMarker){
      foundSettingsString = true;
    }    

  }

}  //end getConfigFromBuffer

If you recall, my serial buffer being passed in contains the following:

AT+CIPSEND=74

GET /methods/v1/getsensorconfig?keyId=c35ba5da9cda156c251dad6c89b2

SEND OK

+IPD,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422206666117
OK

OK
Unlink

The output of the function gives me this:

Inside getConfigFromBuffer function
Dumping configBuffer....
136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=k

I am not sure what endMarker to use. According to this reference string - Arduino Reference a '\0' should get the entire line for me and not truncate the time.now= value.

Any ideas?

Stephen

11 
char startMarker = '+IPD,';

Question : how many characters can you fit in a char variable ?

Even if the answer were 5 then how would this work comparing 1 character with 5 ?

   else if (buffer[i] == startMarker){
12

+IPD,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422206666117
OK

If this is what is printed out from a serial capture, then it appears that the data string is terminated with a carriage return/line feed (note that the OK is on a new line). I'd probably use the : as the capture start marker and the carriage return as the end of data marker. Once that data string is captured, you can use parsing functions to capture the data between each set of = and ,.

13

sfyffe:
If you recall, my serial buffer being passed in contains the following:

AT+CIPSEND=74

GET /methods/v1/getsensorconfig?keyId=c35ba5da9cda156c251dad6c89b2

SEND OK

+IPD,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422206666117
OK

OK
Unlink

I'm confused. Do you mean that your Arduino receives all of this (I have deliberately removed the line breaks)

AT+CIPSEND=74 > GET /methods/v1/getsensorconfig?keyId=c35ba5da9cda156c251dad6c89b2 SEND OK +IPD,136:current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=1422206666117 OK OK Unlink

And if that is correct, what are you trying to extract from it?

And another question - do you need to receive all of it or would it be sufficient to receive part of it and discard the remainder. ?

...R

14

Hi,

You are correct this would not work well at all.. Changing my start marker and end marker to the following:

char startMarker = ':';
char endMarker = '\r';

had a little better results but still truncating the ending.

Here is the output:

current.enabled=true,led.enabled=true,led.frequency=300,temp.enabled=false,temp.frequency=60,time.frequency=86400,time.now=14224

ZoomKat, yes according to the result window of RealTerm there is a CRLF sequence at the end of this string. I have tested with '\r' and '\r\n', as well as ' ' and result are about the same with not capturing the full line.

UPDATE:

To your second post. Yes, my Arduino code receives all of this from the ESP8266 after it connects to a web service to get these configuration settings. I am only after the line beginning with +IPD. I have a key value pair processing routine that I will use to setup the sensor with.

Stephen

15

I am only after the line beginning with +IPD

You need to understand that the arduino only does single character comparisons, not string comparisons. You will need to write code or use code that can search for character strings in the streaming data. Textfinder below might be of use. Bottom is some weather code that extracts desired data from a weather web page. The loss of the final characters might be due to insufficient memory to capture the total string.

http://playground.arduino.cc/Code/TextFinder

// Include description files for other libraries used (if any)
#include <SPI.h>
#include <Ethernet.h>

// Define Constants
// Max string length may have to be adjusted depending on data to be extracted
#define MAX_STRING_LEN  20

// Setup vars
char tagStr[MAX_STRING_LEN] = "";
char dataStr[MAX_STRING_LEN] = "";
char tmpStr[MAX_STRING_LEN] = "";
char endTag[3] = {'<', '/', '\0'};
int len;

// Flags to differentiate XML tags from document elements (ie. data)
boolean tagFlag = false;
boolean dataFlag = false;

// Ethernet vars
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 113, 200 }; // www.weather.gov

// Start ethernet client
EthernetClient client;

void setup()
{
  Serial.begin(9600);
  Serial.println("Starting WebWx");
  Serial.println("connecting...");
  Ethernet.begin(mac, ip);
  delay(1000);

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
    client.println();
    delay(2000);
  } else {
    Serial.println("connection failed");
  }  
}

void loop() {

  // Read serial data in from web:
  while (client.available()) {
    serialEvent();
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("Disconnected");
    Serial.println("==================================");
    Serial.println("");
    client.stop();

    // Time until next update
    //Serial.println("Waiting");
    for (int t = 1; t <= 15; t++) {
      delay(60000); // 1 minute
    }

    if (client.connect(server, 80)) {
      //Serial.println("Reconnected");
      client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
      client.println();
      delay(2000);
    } else {
      Serial.println("Reconnect failed");
    }      
  }
}

// Process each char from web
void serialEvent() {

   // Read a char
	 char inChar = client.read();
   //Serial.print(".");
  
   if (inChar == '<') {
      addChar(inChar, tmpStr);
      tagFlag = true;
      dataFlag = false;

   } else if (inChar == '>') {
      addChar(inChar, tmpStr);

      if (tagFlag) {      
         strncpy(tagStr, tmpStr, strlen(tmpStr)+1);
      }

      // Clear tmp
      clearStr(tmpStr);

      tagFlag = false;
      dataFlag = true;      
      
   } else if (inChar != 10) {
      if (tagFlag) {
         // Add tag char to string
         addChar(inChar, tmpStr);

         // Check for </XML> end tag, ignore it
         if ( tagFlag && strcmp(tmpStr, endTag) == 0 ) {
            clearStr(tmpStr);
            tagFlag = false;
            dataFlag = false;
         }
      }
      
      if (dataFlag) {
         // Add data char to string
         addChar(inChar, dataStr);
      }
   }  
  
   // If a LF, process the line
   if (inChar == 10 ) {

/*
      Serial.print("tagStr: ");
      Serial.println(tagStr);
      Serial.print("dataStr: ");
      Serial.println(dataStr);
*/

      // Find specific tags and print data
      if (matchTag("<temp_f>")) {
	      Serial.print("Temp: ");
         Serial.print(dataStr);
      }
      if (matchTag("<relative_humidity>")) {
	      Serial.print(", Humidity: ");
         Serial.print(dataStr);
      }
      if (matchTag("<pressure_in>")) {
	      Serial.print(", Pressure: ");
         Serial.print(dataStr);
         Serial.println("");
      }

      // Clear all strings
      clearStr(tmpStr);
      clearStr(tagStr);
      clearStr(dataStr);

      // Clear Flags
      tagFlag = false;
      dataFlag = false;
   }
}

/////////////////////
// Other Functions //
/////////////////////

// Function to clear a string
void clearStr (char* str) {
   int len = strlen(str);
   for (int c = 0; c < len; c++) {
      str[c] = 0;
   }
}

//Function to add a char to a string and check its length
void addChar (char ch, char* str) {
   char *tagMsg  = "<TRUNCATED_TAG>";
   char *dataMsg = "-TRUNCATED_DATA-";

   // Check the max size of the string to make sure it doesn't grow too
   // big.  If string is beyond MAX_STRING_LEN assume it is unimportant
   // and replace it with a warning message.
   if (strlen(str) > MAX_STRING_LEN - 2) {
      if (tagFlag) {
         clearStr(tagStr);
         strcpy(tagStr,tagMsg);
      }
      if (dataFlag) {
         clearStr(dataStr);
         strcpy(dataStr,dataMsg);
      }

      // Clear the temp buffer and flags to stop current processing
      clearStr(tmpStr);
      tagFlag = false;
      dataFlag = false;

   } else {
      // Add char to string
      str[strlen(str)] = ch;
   }
}

// Function to check the current tag for a specific string
boolean matchTag (char* searchTag) {
   if ( strcmp(tagStr, searchTag) == 0 ) {
      return true;
   } else {
      return false;
   }
}
16

Thank You, Increasing the memory size a bit got all of the characters. I will just key on the + for now.

I will get my Ethernet shield out and test this code tonight. Being able to handle full strings will be very helpful.

Best Regards,

Stephen

17

the arduino only does single character comparisons, not string comparisons

Is that really the case ? What about functions such as strstr() ?

18

sfyffe:
Yes, my Arduino code receives all of this from the ESP8266 after it connects to a web service to get these configuration settings. I am only after the line beginning with +IPD.

So why not use + as the start marker and linefeed (or carriage return) as the end marker in the 3rd example in serial input basics

...R

19

Using the + and carriage return is what I am doing in my final code. With this specific GET call I know that I will always and only get a + at the beginning of the line I want.

Another alternative for someone in control of the back end is to simply send out your own start marker and end markers that way there is guarantee.

This works well...

Stephen

20

UKHeliBob:
k that
Is that really the case ? What about functions such as strstr() ?

As best as I know, the arduino can only read and evaluate one character at a time from a serial character stream. Are you implying that strstr() can be used to look for a specific character string in the serial character stream?