Fast dynamic/inconsistent delimited serial data integer parser to int array?

[UPDATE] I have just added a video of the latest Python to Arduino parser that shows Python controlling an RGB LED on the Arduino with a pretty decent update interval. I have also added two python scripts along with a compatible arduino sketch for anyone who is interested.

Implemented into one of the python scripts now is a stress testing portion for really testing a wide verity of delimiters and format styles while getting to see good visual feedback from the RGB LED.

[/UPDATE]

Hi,

im having a little trouble extracting some information from a string.

I have attempted to build a few sketches previously but did not like their behavior, so im starting from scratch and hoping someone here might be able to help.

basically i have a few Dri-Air ARID-X10D dryer units, and they have on-board rs232 that i am attempting to read from.

there are multiple ways to extract data from the unit, the simplest is to enable "verbose mode" and every 10 seconds the unit will transmit a CR terminated string (no handshaking involved). the string looks like this:

"97, 97, 97, 91, 91, 91, 82, 81,235, 82, 82, 82, 82, 68, 47,-46 195 192 194 851 0 107 18 0 0 0 0\n"

which to me looks very inconsistent (some values are separated by commas, some by spaces, some by double spaces and some by comma/space). i have no idea what the engineers where thinking, but am fully confident the Arduino platform is up to the challenge. but for a beginner in C++ like me this seems like a nightmare scenario!

initially what i wanted to do was take that string, parse through it and fill an integer array with the values present in that string. for example:

arry[0] = 97 arry[1] = 97 arry[2] = 97 arry[3] = 91 arry[4] = 91 arry[5] = 91 arry[6] = 82 arry[7] = 81 arry[8] = 235 and so on..

but i am unaware as to the existence of any libraries available that can do this directly (i suppose something similar to regular expressions would suffice), or if there are functions that i might not be aware of that would make this operations a little easier?

Thank you all so much for any advice!!!

Have a look at http://arduino.cc/en/Serial/ParseInt

I think Serial.parseint() will work for you.

That was exactly what i was looking for!

except for one little thing that i cant seem to get past.

if i send the string: "0\n" parseInt will return an integer 0, however if i send the string: "foo\n" parseInt (after timing out) will still return an integer 0.

#define BUF_SIZE 256
#define LED_GRN 3

char buffer[BUF_SIZE];

void setup() {
 Serial.begin(9600); 
 pinMode(LED_GRN, OUTPUT);
}

void loop() {
  if (Serial.available() > 0) {
    int brightness = Serial.parseInt();
    Serial.println(brightness);
    if (brightness != 0) { 
            //This means that i could never set brightness to zero but is required to avoid corruption from time outs
            //as the timeout also returns a zero but would not be accurate..
      analogWrite(LED_GRN,constrain(brightness,0,255));
    } 
    while (Serial.read() != -1); // a working alternative to Serial.flush();
  }
}

would it be wise to maybe add a timer in there that would be reset every time just before entering the parseInt() function and is them compared in the If statement to see if a timeout condition occurred?? eg:

int timeout_length = 1000; void loop() { int some_timer = millis(); Serial.parseInt();

if (some_timer + timeout_length >= millis()) { //Then apply the brightness or anything you wish, else an actual timeout did occur.. } }

I ask this because i am worried about millis() "rollover" conditions and its possible effects.

If the number of fields is constant and the separators are known then you could use sscanf() to parse the whole string. Alternatively you could use strtok to extract the individual fields and then use atol() to convert them from decimal strings to numbers - this approach would be particularly useful where the number of fields was not fixed.

Thanks for the help!

i actually wrote this and i think it pretty much does with i need it to do. but its late and im at home so i wont be able to try it out till tomorrow…

though anyone can test it by connecting an RGB led to pins 3,5 and 6.

then in a terminal you can enter strings like:

“255,0,0\n” // would equal RED (255,0,0).
“0a 88cc c0\n” // would be a faint GREEN (0,88,0).
“hello0mynameis199megaBlocks199\n” // would be a dim TURQUOISE (0,199,199).

the color values get stored in the first three elements of the “values” array. Many more could be specified and i set the the max in this example to 24.

it seems to be pretty forgiving as to which delimiters are used (or even how many are used). it just extracts entire integer representations (negative values are handled) from the ascii characters coming in from serial then after going through atoi() it just stacks them into the values array until a carriage return is detected. i havnt found it to block or notice any long delays. but numbers over 32767 are not supported and do cause problems…

#define BUF_SIZE 256
#define LED_RED 3
#define LED_GRN 5
#define LED_BLU 6

char buffer[BUF_SIZE];
int bufIdx = 0;
int values[24];
int response_pos = 0;
char serin;
int x;
void setup() {
 Serial.begin(38400); 
 pinMode(LED_GRN, OUTPUT);
}

void loop() {
  if (Serial.available() > 0) {
    serin = Serial.read();
    if (serin >= 48 && serin <= 57 || (bufIdx == 0 && serin == 45)) {
        buffer[bufIdx] = serin;
        bufIdx += 1;
    } else if (serin == 13) {
        parse_buffer();
        print_all_vals();
        response_pos = 0;
        clear_buffer();
    } else if (bufIdx != 0) {
        parse_buffer();
        
    }
  }
}

void print_all_vals() {
   for (x = 0; x < response_pos; x++) {
        Serial.println(values[x]);
   }
   analogWrite(LED_RED,values[0]);
   analogWrite(LED_GRN,values[1]);
   analogWrite(LED_BLU,values[2]);
}

void parse_buffer() {
  values[response_pos] = atoi(buffer);
  response_pos += 1;
  //Serial.println(values[0]);
  clear_buffer();
}

void clear_buffer() {
  for (x = 0; x <= BUF_SIZE; x++) {
    buffer[x] = 0;
  } 
  bufIdx = 0;
}

This next bit of code is pure Python and it interacts with this sketch quite nicely. the only thing that needs to be ensured is that you configure the port=“COMx” correctly.
when run the RGB led will fade through many different colors very quickly as it calculates the cosine of the variable x in its main for loop.
its quite simple but was good for me to test the above parser’s behavior.

import serial, time, math

ser = serial.Serial(port="COM16", baudrate=38400, timeout=5)

try:
    print "please wait for port init"
    time.sleep(5)
    for x in range(10000):
        speed = 100 #greater is slower
        fx = float(x)/speed
        
        R_val = 2*0.0
        R = int((math.cos(float((fx)+R_val))+1)*128)
        
        G_val = 2*0.5
        G = int((math.cos(float((fx)+G_val))+1)*128)
        
        B_val = 2*1.0
        B = int((math.cos(float((fx)+B_val))+1)*128)
        output = "%s,%s,%s" % (R,G,B)
        output += chr(13)
        print output
        #if ser.inWaiting > 0:
          #serin = ser.read(1)
          #print serin  
        ser.flushInput()
        time.sleep(0.005)
        ser.write(output)        
finally:
    ser.close()
    print "Ports Closed"

This is an updated version of the aforementioned Python code. simply run it the same way as the last one except there are 3 controls you can use:

ENABLE_STRESS_TESTING ENABLE_RETURN_DATA_DISPLAY ENABLE_SENT_DATA_DISPLAY

And one config parameter:

COMPORT="COMx"

in this version i added a stress testing mechanism to see if there are any hiccups in the LED's performance as its giving me visual feedback. But the LED's color transitions (even with the mess of delimiters) is still very "fluid" with no fluctuation that i have observed.

Note: because of how the arduino is currently programmed to interpret and accept negative values; if you send delimiters consisting of the "-" characters it can cause major problems, this is the only character i found which you need to be specific in its ordering and should only be used preceding a number in which you plan to indicate as a negative value.

Also note that this was built on Python 2.7.3 using the open source PySerial module. though earlier versions may support it, i have not ported it for Py3.x.

To enable or disable stress testing change the boolean True/False condition on ENABLE_STRESS_TESTING

Python 2.7 Script:

#!/usr/bin/env python

"""serialIntParser-With-Delimiter_Strss_Test.py: Tests arduinos using high speed serial parser."""

__author__  = "Grant Olsen"
__copyright__ = "Copyright (C) 2013 Grant Olsen"
__license__ = "Public Domain"
__version__ = "2.0"

import serial, time, math

COMPORT="COM16"

ENABLE_STRESS_TESTING = True #generates a large verity of different delimiter and widths.
ENABLE_RETURN_DATA_DISPLAY = False #prints out the data coming back from the arduino.
ENABLE_SENT_DATA_DISPLAY = True #prints out the string of data being sent to the arduino.

StressTestList = ["%s,%s,%s", \
              "%s./!#$^&*()_+%s./!#$^&*()_+%s./!#$^&*()_+", \
              "abcdefg%shijklmnop%sqrstu%spwxyz", \
              "////////////%s////////////%s////////////%s////////////", \
              "\\\\\\\\\\\\%s||||||||||||%s////////////%s............"]

try:
    print "Please wait for port to initialize.\n\n\n"
    ser = serial.Serial(port=COMPORT, baudrate=38400, timeout=5)
    time.sleep(5)
    listNow = 0
    for x in range(100000):
        speed = 100 #greater is slower aka higher resolution
        fx = float(x)/speed
        
        R_val = 2*0.0
        R = int((math.cos(float((fx)+R_val))+1)*128)
        
        G_val = 2*0.5
        G = int((math.cos(float((fx)+G_val))+1)*128)
        
        B_val = 2*1.0
        B = int((math.cos(float((fx)+B_val))+1)*128)
        if ENABLE_STRESS_TESTING == True:#generate a bunch of delimiters
            formatStr = StressTestList[listNow]
        else: 
            formatStr = "%s,%s,%s"   
        listNow += 1
        if listNow >= len(StressTestList): listNow = 0
        output =  formatStr % (R,G,B)
        output += chr(13)
        if ENABLE_SENT_DATA_DISPLAY:
            print output 
        if ENABLE_RETURN_DATA_DISPLAY: 
            if ser.inWaiting > 0:
              serin = ser.read(ser.inWaiting())
              print serin, #this part needs some work
        ser.flushInput()
        time.sleep(0.005)
        ser.write(output)
except Exception as e:
    print e
finally:
    ser.close()
    print "\nPort Closed.\n"

I just hope someone finds this useful someday! :)