In need of a way to parse a very Very VERY long string and extract some values??

Hi everyone!

i have some machines here that have an RS232 interface that i can use to poll for status information.

problem is, the engineers who built the dang thing were not thinking "simple" when they developed their protocol. instead it seems they were thinking along the lines of "human readable" output...

so to see the value of the current temperature i have to issue a "STATUS" command and the device returns a stream of data that's roughly 1430 characters long.

then find the line:

"Process Temps 175 , 77 Degrees"
-or-
"Process 1 setpoint = 180 - actual temp = 175 Temp Alarm Delta = 50"

<<>>

I have been thinking of some ways to sift through this "congested string" and was considering buffering everything up (i know.. bad) then using something like Nick Gammon's regular expression library to extract the values i need. (not sure how fast that would be either, but the polling rate could be low and every 2-10 seconds would be great..)
as a note: the baudrate from the device is set to 38400. and there is no easy way to change it.

I was curious how you guys would go about parsing this efficiently??

Thanks!!!!

Response from unit after issuing "STATUS" command:

Dryer Serial =  D18885

Temperatures are in F

Two Hopper System

Motor Selected

Hopper 1 enabled

System Autostart Enabled

Loop Break Alarm DISABLED

Current Test DISABLED

Thermocouple Test ENABLED

Installed thermocouples = fff

Process 1 Temp Offset = +0

Process 2 Temp Offset = +0

Dewpoint Offset = +0

Drying Trip point = 330

Cooling Trip point = 280

Regen Setpoint = 600

Heat Exchanger Hi Temp Limit = 180,

Heat Exchanger Low Temp Limit = 170,

Preheat Time = 60

Process 1 setpoint = 180 - actual temp = 175 Temp Alarm Delta = 50

Process 2 setpoint = 180 - actual temp = 77 Temp Alarm Delta = 50

Low Temp Alarm Disabled

Dewpoint Alarm Enabled

Dewpoint: -53C  No Limit Dewpoint: -53C

Dewpoint Alarm Level = 25

Loader On Time = 5

Loader Delay = 15

Zone 2 heating - Mode: Temperature



Single Hopper Cool Trip Delta = 75

Heater Difference = 75

Network Address = 0

LEARNED HEATER CURRENTS

  Z1B  Z1T  Z2B  Z2T  HP1  HP2  OFF

  0.1  0.1  0.1  0.0  0.0  0.0  0.0

 Process 1 Setback is OFF

 Process 2 Setback is OFF

Setback Inhibit Times         120 ,  120  minutes

Setback Activate Delta Temps  75 , 75 Degrees

Setback Temp Delta            30 , 30  Degrees

Setback Restore Temps         100 , 100  Degrees

Process Temps                 175 , 77  Degrees

Process Return Temps          77 , 77  Degrees

Inhibit Timers                3 , 3  minutes

Setback Idle Times            30 , 30  minutes

If you get the whole of that every time I suggest you scan the received characters for the character at the start of each line - they seem to have a limited number of options. Then for the duplicates you could scan the second character.

...R

Seems that the data strings are separated by two sets of carriage return/line feeds, which might could be used as data packet delimiters.

I don't have my all of my code handy to show you an example but here is how I do it.

Read in the text and append it to a string called current line, when you hit a new line '\n' then you you know you have a full line of text

char myChar;
    // read from the file until there's nothing else in it:
    while (webFile.available()) {
      myChar = webFile.read(); 
      currentLine += myChar;
if (myChar == '\n') {  //new line
texttofind = "Loader Delay = ";
        if (texttofind == left(currentLine,1,len(texttofind)) {
        //parse the info for the result
       }
texttofind = "Process Temps "
 if (texttofind == left(currentLine,1,len(texttofind)) {
        //parse the info for the result
       }

// all done with the currentline read in the next full line
currentLine =""
     }

If the string is too long to be processed in memory then I'd use a finite state machine to record progress through the expected sequence of tokens, and within states where you're waiting for a constant expected string to arrive just keep the expected string in PROGMEM and have a counter to record how many characters in that string have been matched so far against the incoming string. For tokens that you wanted to extract from the string, you'd do this character by character as they arrive.

With that approach you'd use a bit of PROGMEM, and whatever data you needed to receive values extracted from the incoming string, and a few bytes worth of state variables, but you wouldn't have to buffer the raw string.

Robin2:
If you get the whole of that every time I suggest you scan the received characters for the character at the start of each line - they seem to have a limited number of options. Then for the duplicates you could scan the second character.

...R

This is just what i needed and gave me some ideas and i actually got something working (@Robin2, i hope this is kind of along the lines of what you were suggesting :slight_smile: ). And it may be ugly but so far my Python test simulation script gets good and continuous results (and uses WAY more bandwidth then the devices will actually ever see while in service).
ill post it here for posterity (and of course my records)

Arduino Sketch

#include <avr/pgmspace.h>
prog_char string_0[] PROGMEM = "Process Temps";
prog_char string_1[] PROGMEM = "Dewpoint:";
prog_char string_2[] PROGMEM = "String 2";
prog_char string_3[] PROGMEM = "String 3";
prog_char string_4[] PROGMEM = "String 4";
prog_char string_5[] PROGMEM = "String 5";


// Then set up a table to refer to your strings.

PROGMEM const char *string_table[] = 	   // change "string_table" name to suit
{   
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  string_5 };
  
const int bufSize = 64;
char cpy_buffer[bufSize+1];    // make sure this is large enough for the largest string it must hold
char lin_buffer[bufSize+1];
int lin_buffer_idx = 0;

void setup()			  
{
  Serial.begin(38400);
}

void clearBuffer() {
  memset(lin_buffer,0,sizeof(lin_buffer));
  lin_buffer_idx = 0;
}


int readInLine() {
  char input = Serial.read();
  if (input == '\r' || input == '\n') {
   return 1;
  } else if (input != -1) {
   lin_buffer[lin_buffer_idx] = input;
   lin_buffer_idx++;
   lin_buffer_idx = constrain(lin_buffer_idx, 0, bufSize-1);
   return 0;
  }  
  return -1;
}

void loop()			  
{
  if (readInLine() == 1) {
    for (int i = 0; i < 6; i++)
    {
      strcpy_P(cpy_buffer, (char*)pgm_read_word(&(string_table[i]))); // Necessary casts and dereferencing, just copy. 
      boolean checker = 1;
      for (int x = 0; x < bufSize; x++) {
        if ( int(cpy_buffer[x] ) == 0 ) { //fires once the string in PORGMEM reaches a /0 null
          break;
        }
        checker &= (cpy_buffer[x] == lin_buffer[x]);
      }
      
      if (checker == 1) {
        //apply value to variable here!!!
        Serial.println(lin_buffer);
        break;
      } 
    }

    clearBuffer(); //always do this last.
  }
}

Python Script

import serial, time

text = """

Dryer Serial =  D18885

Temperatures are in F

Two Hopper System

Motor Selected

Hopper 1 enabled

System Autostart Enabled

Loop Break Alarm DISABLED

Current Test DISABLED

Thermocouple Test ENABLED

Installed thermocouples = fff

Process 1 Temp Offset = +0

Process 2 Temp Offset = +0

Dewpoint Offset = +0

Drying Trip point = 330

Cooling Trip point = 280

Regen Setpoint = 600

Heat Exchanger Hi Temp Limit = 180,

Heat Exchanger Low Temp Limit = 170,

Preheat Time = 60

Process 1 setpoint = 180 - actual temp = 190 Temp Alarm Delta = 50

Process 2 setpoint = 180 - actual temp = 78 Temp Alarm Delta = 50

Low Temp Alarm Disabled

Dewpoint Alarm Enabled

Dewpoint: -53C  No Limit Dewpoint: -53C

Dewpoint Alarm Level = 25

Loader On Time = 5

Loader Delay = 15

Zone 1 heating - Mode: Temperature



Single Hopper Cool Trip Delta = 75

Heater Difference = 75

Network Address = 0

LEARNED HEATER CURRENTS

  Z1B  Z1T  Z2B  Z2T  HP1  HP2  OFF

  0.1  0.1  0.1  0.0  0.0  0.0  0.0

 Process 1 Setback is OFF

 Process 2 Setback is OFF

Setback Inhibit Times         120 ,  120  minutes

Setback Activate Delta Temps  75 , 75 Degrees

Setback Temp Delta            30 , 30  Degrees

Setback Restore Temps         100 , 100  Degrees

Process Temps                 190 , 78  Degrees

Process Return Temps          78 , 78  Degrees

Inhibit Timers                95 , 95  minutes

Setback Idle Times            30 , 30  minutes
"""

try:
    ser = serial.Serial(timeout=0.25, baudrate=38400, port="COM75")
    time.sleep(2)
    while True:
        for c in text:
            ser.write(c)
            #time.sleep(0.1)
            if ser.inWaiting() > 0:
                for x in xrange(ser.inWaiting()):
                    print ser.read(1),


finally:
    ser.close()

Python's Output

P r o c e s s   T e m p s                                   1 9 0   ,   7 8     D e g r e e s 

D e w p o i n t :   - 5 3 C     N o   L i m i t   D e w p o i n t :   - 5 3 C 

This Repeats Forever!!

Thanks Everyone!!!

I haven't studied your sketch carefully.

I was just thinking of saving the initial letters (P D D N in your example) and waiting until a newline is seen then checking is the next char the next one of these.

Thinking further about it, perhaps all you need to do is count the number of newlines to know where you are and then use the first one or 2 chars as confirmation.

Once you are at the start of the line you want, just save everything until the next newline.

Parse the line then or later.

By the way "contentious" is NOT the right word - it means controversial. Also, I think you meant that it uses less bandwith (not more) if you mean there is plenty of room for more stuff.

...R

@Robin,
that was actually a typo, I meant to say continuous results as I had an infinite loop running.
Also I should have probably said that my test use more of the available bandwidth than in the real world. So I could turn down the polling rate and effectively decreasing processor overhead in production.

That last example was just of the core "matching" element in the parser, and the python script was very simple.
So i used a little of my weekend to add some handshaking and ascii-to-integer conversion for the stored results.

So far, tests have been passing without a hitch; again, so far.....
even by copying and pasting two of those fake status messages together and sending, it still gets parsed ok; its around three times where things get too large for the loop timings i have set, which is easily adjusted by changing the "onTime" variable in the arduino sketch.

though i am very interested in feedback from anyone who runs the test; i also hope this could be of use for someone in the future.

and thank you everyone for your help!!

Enjoy! :slight_smile:

SUPER LONG STRING PARSER - ARDUINO SKETCH:

#include <avr/pgmspace.h>
prog_char string_0[] PROGMEM = "Process Temps";
prog_char string_1[] PROGMEM = "Dewpoint:";
prog_char string_2[] PROGMEM = "String 2";
prog_char string_3[] PROGMEM = "String 3";
prog_char string_4[] PROGMEM = "String 4";
prog_char string_5[] PROGMEM = "String 5";

// Then set up a table to refer to your strings.

PROGMEM const char *string_table[] = 	   // change "string_table" name to suit
{   
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  string_5 };
const int TotalNumOfValues = 6;
int VALUES[TotalNumOfValues];

const int bufSize = 64;
char cpy_buffer[bufSize+1];    // make sure this is large enough for the largest string it must hold
char lin_buffer[bufSize+1];
int lin_buffer_idx = 0;

void setup() {
  Serial.begin(38400);
}

void printValues() {
  Serial.println("VALUES:");
  for (int i = 0; i < TotalNumOfValues; i++) {
    Serial.println( VALUES[i] ); 
  }
}

void clearBuffer() {
  memset(lin_buffer,0,sizeof(lin_buffer));
  lin_buffer_idx = 0;
}

int readInLine() {
  char input = Serial.read();
  if (input == '\r' || input == '\n') {
   return 1;
  } else if (input != -1) {
   lin_buffer[lin_buffer_idx] = input;
   lin_buffer_idx++;
   lin_buffer_idx = constrain(lin_buffer_idx, 0, bufSize-1);
   return 0;
  }  
  return -1;
}

void parseIncoming() {
  if (readInLine() == 1) {
    int i = 0;
    for (i; i < 6; i++)
    {
      strcpy_P(cpy_buffer, (char*)pgm_read_word(&(string_table[i]))); // Necessary casts and dereferencing, just copy. 
      boolean checker = 1;
      int x = 0;
      for (x; x < bufSize; x++) {
        if ( int(cpy_buffer[x] ) == 0 ) { //fires once the string in PORGMEM reaches a /0 null
          break;
        }
        checker &= (cpy_buffer[x] == lin_buffer[x]);
      }
      if (checker == 1) {
        //now that we have a match lets scan it and pluck out the first set of numbers, 
        int numBufferIdx = 0;
        char numBuffer[8] = {'\0'};
        boolean numFlag = true; //goes false to indicate when the first digit was seen
        for (int c = x; c <= bufSize; c++) {
          numFlag &= (  ( !(isdigit(lin_buffer[c]) ) && ( !(lin_buffer[c] == '-')) )  );  //will go false if digit or '-' is seen then is anded to the flag.
          if (!numFlag) { //after this sees digit, if it sees anything but digit it moves on.
            numBuffer[numBufferIdx] = lin_buffer[c];
            numBufferIdx++;
            if ( !isdigit(lin_buffer[c+1]) ) { //saw something other than digit, moving on...
              VALUES[i] = atoi(numBuffer);//APPLYING VALUE TO VARIABLE HERE
              break; //done, lets break out and continue checking input for matches
            }
            numBufferIdx = constrain(numBufferIdx, 0, 7);
          }
        }
        break;
      } 
    }
    clearBuffer(); //always do this last.
  }  
}  

long previousMillis = 0; 
long interval = 2000;
long onTime = 1000;
boolean poll = true;

void loop() {
  unsigned long currentMillis = millis();
  if(currentMillis - previousMillis > interval) {
    //do communication opperations here
    if (poll) {
      Serial.print("STATUS\r\n");
      poll = false;
    }
    parseIncoming(); 
    if(currentMillis - previousMillis > interval + onTime) {
      //do one time opperation here
      poll = true;
      previousMillis = currentMillis; 
      Serial.print("REPLY\r\n"); 
      printValues();
    }
  } else {
    //do long bulky opperations here
  } 
  //do short critical stuff here
}

DEVICE SIMULATOR - PYTHON SCRIPT:

import serial, time

#Text Of The Fake Response:
text = """

Dryer Serial =  D18885

Temperatures are in F

Two Hopper System

Motor Selected

Hopper 1 enabled

System Autostart Enabled

Loop Break Alarm DISABLED

Current Test DISABLED

Thermocouple Test ENABLED

Installed thermocouples = fff

Process 1 Temp Offset = +0

Process 2 Temp Offset = +0

Dewpoint Offset = +0

Drying Trip point = 330

Cooling Trip point = 280

Regen Setpoint = 600

Heat Exchanger Hi Temp Limit = 180,

Heat Exchanger Low Temp Limit = 170,

Preheat Time = 60

Process 1 setpoint = 180 - actual temp = 190 Temp Alarm Delta = 50

Process 2 setpoint = 180 - actual temp = 78 Temp Alarm Delta = 50

Low Temp Alarm Disabled

Dewpoint Alarm Enabled

Dewpoint: -53C  No Limit Dewpoint: -53C

Dewpoint Alarm Level = 25

Loader On Time = 5

Loader Delay = 15

Zone 1 heating - Mode: Temperature



Single Hopper Cool Trip Delta = 75

Heater Difference = 75

Network Address = 0

LEARNED HEATER CURRENTS

  Z1B  Z1T  Z2B  Z2T  HP1  HP2  OFF

  0.1  0.1  0.1  0.0  0.0  0.0  0.0

 Process 1 Setback is OFF

 Process 2 Setback is OFF

Setback Inhibit Times         120 ,  120  minutes

Setback Activate Delta Temps  75 , 75 Degrees

Setback Temp Delta            30 , 30  Degrees

Setback Restore Temps         100 , 100  Degrees

Process Temps                 190 , 78  Degrees

Process Return Temps          78 , 78  Degrees

Inhibit Timers                95 , 95  minutes

Setback Idle Times            30 , 30  minutes

"""

BUFFER = ''

try:
    ser = serial.Serial(timeout=0.25, baudrate=38400, port="COM32")
    time.sleep(2)
    ser.flushInput();
    while True:
        while ser.inWaiting() > 0:
            D = ser.read(1)
            if D == '\r' or D == '\n':
                if BUFFER == "STATUS":
                    for c in text:
                        ser.write(c)
                if BUFFER == "REPLY":
                    linebuff = ''
                    time.sleep(0.1)
                    while ser.inWaiting() > 0:
                        linebuff += ser.read(1)
                        
                    print linebuff #the returned values     
                    
                    if linebuff <> '\nVALUES:\r\n190\r\n-53\r\n0\r\n0\r\n0\r\n0\r\n': #change this when you make changes, hint: print repr(linebuff)
                        PassFail = "FAIL!!!! " + repr(linebuff)
                    else:   
                        PassFail = "PASS! :-)"
                    print PassFail
                BUFFER = '';
                ser.flushInput();
            else:
                BUFFER += D


finally:
    ser.close()

OUTPUT OF THE ABOVE EXAMPLE:

>>>
VALUES:
190
-53
0
0
0
0

PASS! :-)

Ok, this will probably be my last update to this code for this post.

This next sketch allows for use of something similar to a wild card using the * character. If inserted after the search string the parser will be instructed to not extract the numerical values but instead just extract the entire string after the match.

so for example:
search string = "Dryer Serial ="
string from the device = "Dryer Serial = D18885"
result = 18885

search string = "Dryer Serial =*"
string from the device = "Dryer Serial = D18885"
result = D18885

notes:

  • this is compatible with the above Python test script.
  • the variable "values_buffer_size" sets how large the array which houses the results can be eg. "D18885" would fit in 6 but should always be set for the largest value you wish to capture.
  • the variable "bufSize" should be set larger than the largest line to be read in. eg. if i were trying to get the value from "Heat Exchanger Hi Temp Limit = 180," i would have to set this to greater than 35.
  • Binary sketch size: 2,882 bytes.

Enjoy!

SUPER LONG STRING PARSER (VER-2) - ARDUINO SKETCH:

#include <avr/pgmspace.h>
prog_char string_0[] PROGMEM = "Process Temps";
prog_char string_1[] PROGMEM = "Dewpoint:";
prog_char string_2[] PROGMEM = "Preheat Time";
prog_char string_3[] PROGMEM = "Dewpoint Offset =*";
prog_char string_4[] PROGMEM = "Hopper 1*";
prog_char string_5[] PROGMEM = "Dryer Serial =*";
prog_char string_6[] PROGMEM = "Current Test*";

// Then set up a table to refer to your strings.

PROGMEM const char *string_table[] = 	   // change "string_table" name to suit
{   
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  string_5,
  string_6,
  };
  
const int TotalNumOfValues = sizeof(string_table)/2;
const int values_buffer_size = 8;
char VALUES[TotalNumOfValues][values_buffer_size];

const int bufSize = 64;
char cpy_buffer[bufSize+1];    // make sure this is large enough for the largest string it must hold
char lin_buffer[bufSize+1];
int lin_buffer_idx = 0;

void setup() {
  Serial.begin(38400);
}

void printValues() {
  Serial.println("VALUES:");
  for (int i = 0; i < TotalNumOfValues; i++) {
    Serial.println( VALUES[i] ); 
  }
}

void clearBuffer() {
  memset(lin_buffer,0,sizeof(lin_buffer));
  lin_buffer_idx = 0;
}

int readInLine() {
  char input = Serial.read();
  if (input == '\r' || input == '\n') {
   return 1;
  } else if (input != -1) {
   lin_buffer[lin_buffer_idx] = input;
   lin_buffer_idx++;
   lin_buffer_idx = constrain(lin_buffer_idx, 0, bufSize-1);
   return 0;
  }
  return -1;
}

void parseIncoming() {
  if (readInLine() == 1) {
    int i = 0;
    for (i; i < TotalNumOfValues; i++) {
      strcpy_P(cpy_buffer, (char*)pgm_read_word(&(string_table[i]))); // Necessary casts and dereferencing, just copy.
      boolean checker = 1;
      int x = 0;
      boolean parseNumbers = true;
      for (x; x < bufSize; x++) {
        if (cpy_buffer[x] == '*') {
          parseNumbers = false; //indicates if we want to extract only the numbers from the text or the entire text after the match.
          break;
        }
        if ( int(cpy_buffer[x] ) == 0 ) { //fires once the string in PORGMEM reaches a /0 null
          break;
        }
        checker &= (cpy_buffer[x] == lin_buffer[x]);

      }
      if (checker == 1) {
        //now that we have a match lets scan it and pluck out the first set of numbers, 
        int numBufferIdx = 0;
        char numBuffer[values_buffer_size] = {'\0'};
        boolean numFlag = true; //goes false to indicate when the first digit was seen
        for (int c = x; c <= bufSize; c++) {
          
          if (parseNumbers == true) {
            
            
            //PARSER1 FOR EXTRACTING NUMERICAL VALUES ONLY.            
            numFlag &= (  ( !(isdigit(lin_buffer[c]) ) && ( !(lin_buffer[c] == '-')) )  );  //will go false if digit or '-' is seen then is anded to the flag.
            if (!numFlag) { //after this sees digit, if it sees anything but digit it moves on.
              numBuffer[numBufferIdx] = lin_buffer[c];
              numBufferIdx++;
              if ( !isdigit(lin_buffer[c+1]) ) { //saw something other than digit, moving on...
                //VALUES[i] = atoi(numBuffer);//APPLYING VALUE TO VARIABLE HERE
                memcpy(VALUES[i], numBuffer, values_buffer_size-1);
                break; //done, lets break out and continue checking input for matches
              }
              numBufferIdx = constrain(numBufferIdx, 0, 7);
            }
            //END PARSER1
            
            
          } else {
            
            
            //PARSER2 FOR EXTRACTING EVERYTHING AFTER THE MATCH STRING (EXCLUDES ANY WHITESPACE CHARACTERS!!!).
            if ( lin_buffer[c] == '\0' ) { 
              memcpy(VALUES[i], numBuffer, values_buffer_size-1);
              break;
            } else if (lin_buffer[c] != ' ') {
              numBuffer[numBufferIdx] = lin_buffer[c];
              numBufferIdx++;
            }
            //END PARSER2
            
            
          }
        }
        break;
      }
    }
    clearBuffer(); //always do this last.
  }
}

long previousMillis = 0; 
long interval = 2000;
long onTime = 1000;
boolean poll = true;

void loop() {
  unsigned long currentMillis = millis();
  if(currentMillis - previousMillis > interval) {
    //do communication opperations here
    if (poll) {
      Serial.print("STATUS\r\n");
      poll = false;
    }
    parseIncoming(); 
    if(currentMillis - previousMillis > interval + onTime) {
      //do one time opperation here
      poll = true;
      previousMillis = currentMillis; 
      Serial.print("REPLY\r\n"); 
      printValues();
    }
  } else {
    //do long bulky opperations here
  } 
  //do short critical stuff here
}

EXAMPLE OUTPUT:

VALUES:
190
-53
60
+0
enabled
D18885
DISABLE