Parsing through data

I am interfacing my ESP8266 with Arduino Uno.
I am sending a GET request using ESP8266 and receive a lot of data through the serial connection(about 1000 characters) .
I want a way to parse through all of this data in order to find the desired string(which lies between 2 other already known strings).
Is there any library for this purpose?
Or is there some program logic?
Also is it possible to create a buffer for taking in all of the data coming through the serial connection and
then going through it.
I am also giving an attachment which shows the data that is received through the connection.
Also if the code is given it will be appreciated.

attach.txt (1.05 KB)

I want a way to parse through all of this data in order to find the desired string(which lies between 2 other already known strings).

Can you please give examples of the known strings and what you are searching for ?

is it possible to create a buffer for taking in all of the data coming through the serial connection and
then going through it.

You could store the incoming data in an array of chars but this will use a lot of memory and may not be feasible in practice.

to save memory, throw away all characters until you get the first character that you need, then you could either:

  1. start storing characters and process it later, or
  2. process it as you go by comparing against the next character you expect, and setting state variables so you remember where you are up to.

Thanks for the reply.
I am specifically looking how to find the strings 20830 and 280 that are towards the end.
Also how to store the characters ?

Subhan95:
Thanks for the reply.
I am specifically looking how to find the strings 20830 and 280 that are towards the end.
Also how to store the characters ?

So each value is between two strings. You need to parse the message until you know you have either leading string and take the value that follows.

You could try to hold a chunk in a ring buffer of say 60 bytes and check for the leading substring at position zero. Then take the value that follows the > and is preceded by <.

Kind of thing.

So do I use a logic of the kind

if(Serial.find(">")
{
//code to take what lies between > and < and store it in an array
//similarly for the second number
}

Also will this create any problems as I heard that the serial buffer can hold 256 bytes of data.

Subhan95:
Also will this create any problems as I heard that the serial buffer can hold 256 bytes of data.

The serial buffer size is only a problem if your code is lazy and allows it to fill up. If you are continuously polling and handling the incoming data, there is no problem.

Not the method I suggested, no.

Collect serial stream into a 60byte buffer and keep checking for the presence of the leading string at position zero. Move the chars in the buffer one position up (effectively deleting the first chat at position zero) and read the next bite into the last position. Testing for the leading string every time you read a new char from the stream into your buffer.

Once you know the buffer contains the leading string, use strtok() to get your number.

I heard that the serial buffer can hold 256 bytes of data.

You heard wrong. It is less than that but it does not matter because you can take data from ot much faster than the serial interface can put it in.

I prefer the solution in reply #2, method 2.

BulldogLowell:
Not the method I suggested, no.

Collect serial stream into a 60byte buffer and keep checking for the presence of the leading string at position zero. Move the chars in the buffer one position up (effectively deleting the first chat at position zero) and read the next bite into the last position. Testing for the leading string every time you read a new char from the stream into your buffer.

Once you know the buffer contains the leading string, use strtok() to get your number.

Just a side-note: a ring (circular) buffer doesn't shift the characters. It keeps an 'in' and an 'out' pointer, and moves around the circle as characters are written at the end, and read from the 'out' index. Its efficiency comes from not having to shift the characters.

Subhan95:
So do I use a logic of the kind

if(Serial.find(">")
{
//code to take what lies between > and < and store it in an array
//similarly for the second number
}

Also will this create any problems as I heard that the serial buffer can hold 256 bytes of data.

So you could try something like this, which works on your test data; though I have not tested beyond that.

That said, I used a version of parseInt() which isnt’ too pretty, but you can get the point on how to capture the data from the stream:

#define BUFF_SIZE 30

const char nixChar = ',';

long viewCount = 0;
long favoriteCount = 0;

char myBuffer[BUFF_SIZE];
int index = 0;
void setup()
{
  Serial.begin(9600);
}
void loop()
{
  if (Serial.available())
  {
    myBuffer[index++] = Serial.read();
    myBuffer[index] = NULL;

    if (strstr(myBuffer, "view-count\">"))
    {
      viewCount = parseNumber(nixChar);
      Serial.print("ViewCount = ");
      Serial.println(viewCount);
      memset(myBuffer, NULL, BUFF_SIZE);
      index = 0;
    }
    else if (strstr(myBuffer, "favorite-count\">"))
    {
      favoriteCount = parseNumber(nixChar);
      Serial.print("FavoriteCount = ");
      Serial.println(favoriteCount);
      memset(myBuffer, NULL, BUFF_SIZE);
      index = 0;
    }
    else if (strstr(myBuffer, "Unlink"))
    {
      Serial.println("Done parsing this chunk.");
      memset(myBuffer, NULL, BUFF_SIZE);
      index = 0;
    }
    if (index > BUFF_SIZE - 1)
    {
      index = BUFF_SIZE - 1;
      for (byte i = 0; i < BUFF_SIZE - 1; i++)
      {
        myBuffer[i] = myBuffer[i + 1];
      }
    }
  }
}

long parseNumber(char skipChar)
{
  boolean isNegative = false;
  long value = 0;
  byte c;
  c = Serial.read();
  if(c < 0)
  {
    return 0;
  }
  do{
    if(c == skipChar)
      ;
    else if(c == '-')
      isNegative = true;
    else if(c >= '0' && c <= '9')
      value = value * 10 + c - '0';
    delay(10);
    c = Serial.read();
  }
  while( (c >= '0' && c <= '9') || c == skipChar );

  if(isNegative)
    value = -value;
  return value;
}

arduinodlb:
Just a side-note: a ring (circular) buffer doesn’t shift the characters. It keeps an ‘in’ and an ‘out’ pointer, and moves around the circle as characters are written at the end, and read from the ‘out’ index. Its efficiency comes from not having to shift the characters.

yeah, I know, I was lazy in my description… maybe “scrolling Buffer” is a better term.

None of the above is fast or efficient, but it may get the OP started.