Parsing serial input

Hey guys.

I have serial input coming from this program called X Sim.

It sends information in values of 0-255, And I can choose how to separate the values. The values appear as ~a1~ for example in the picture below.

I want to split the values and store them in two separate variables, for display on a 7-segment unit.

You need to look up the characters you don't want at asciitable.com & ignore those.

Is it the 'a', '0', '1' and 'a', '0', '2' that you want to show, or the data that will be at those spaces in the string?

CrossRoads:
You need to look up the characters you don't want at asciitable.com & ignore those.

Is it the 'a', '0', '1' and 'a', '0', '2' that you want to show, or the data that will be at those spaces in the string?

The ~a01~ is the data that will be at those spaces in the string.

Okay, same concept then. Receive the characters, and filter for what you want.
If you know that
AB~255~~
arrives first, watch for that, and store the real bytes that follow,
ignore a couple more ~~
and store a couple more.
Then restart and look for AB~255~~ again.

The way it is shown, I would expect you really need to compare like this:
if (Serial.Available)>16 { // 17 bytes are available
if (serialRead() == "A"){
if (serialRead() == "B"){
if (serialRead() == "~"){
if (serialRead() == "2"){
if (serialRead() == "5"){
if (serialRead() == "5"){
if (serialRead() == "~"){
if (serialRead() == "~"){
char1= serialRead(); // will the data always be three characters? Or is it just two?
char2= serialRead();
char3= serialRead();
if (serialRead() == "~"){
if (serialRead() == "~"){
char4= serialRead();
char5= serialRead();
char6= serialRead();
}}}}}}}}}}} // however many are needed to mate with the {s above - use CTRL-T (autoformat) to make it readable

There's much smarter ways to do this, like concatenating all the reads into a string, and then comparing strings.
I'm too tired to work that out right now tho.

CrossRoads:
Okay, same concept then. Receive the characters, and filter for what you want.
...
There's much smarter ways to do this, like concatenating all the reads into a string, and then comparing strings.
I'm too tired to work that out right now tho.

Awesome, Thanks!
I was missing the fact that I need to treat the incoming data as characters. I also found a bit of code on the dead X-Sim forum.
Here's what i'm using now, and I've got it splitting the values nicely!

Hopefully it'll still work when I scale up and have a lot more data coming over the serial connection. But I'll cross that bridge when I come to it!

//X Sim Details//
8-bit Resolution
dezimal output
115200 baud rate

Datapacket with axisinformations:    R~a01~S~a02~
Pause of 25ms
//ARDUINO CODE//

void loop(){
//****************************** READ DATA FROM SERIAL **********************************
  while (Serial.available() > 0){
    kind_of_data = Serial.read();
    if (kind_of_data == 'R' ) Read_Rpm();
    if (kind_of_data == 'S' ) Read_Speed();
  }
}
//****************************** READ DATA FROM SERIAL END ******************************

void Read_Rpm(){

  int RPM = 0;
  delay(2);
  int Rpm100 = Serial.read();
  delay(2);
  int Rpm10 = Serial.read();     //These three lines read in each digit of the 0-255 value.
  delay(2);
  int Rpm1= Serial.read();

//------------------------------Code continues on, converts the values, and displays them. Also controls a rough shift light setup.
}

void Read_Speed(){

  int Speed100 = Serial.read();
  delay(2);
  int Speed10 = Serial.read();    //These three lines read in each digit of the 0-255 value.
  delay(2);
  int Speed1= Serial.read();

//------------------------------Code continues on, converts the values, and displays them.

I've got a few issues now that i'm scaling up to add in a gear indicator.

1st. - Screen lag. I'm not sure if this is due to my baud rate, pause time, or the delay commands i'm using in my read_XYZ() sections.

2nd. - Data resolution. I'd like to have a more accurate display of my data, and the 0-255 is just not enough resolution. You can see a visual example of this in my above video, when the revs are changing slowly.
Can I use the 16-bit option in X Sim and change the Arduino's serial handling code above to match?

Screen lag. I'm not sure if this is due to my baud rate, pause time, or the delay commands i'm using in my read_XYZ() sections.

Yes, it is.

Any of these reasons will cause lag between sending data and the display of that data. Why do you have pauses or delays? Why are you not running at the maximum supported baude rate (115200)?

Display resolution. I'd like to have a more accurate display of my data, and the 0-255 is just not enough resolution. Can I use the 16-bit option in X Sim and change the Arduino code to be compatible?

Yes. Exactly how depends on whether X Sim is sending strings (no changes needed) or binary data (need to read two bytes and do a little (simple) bit shifting and addition to re-create the original value).

In any case, POST YOUR CODE!

CrossRoads:
Okay, same concept then. Receive the characters, and filter for what you want.
If you know that
AB~255~~
arrives first, watch for that, and store the real bytes that follow,
ignore a couple more ~~
and store a couple more.
Then restart and look for AB~255~~ again.

The way it is shown, I would expect you really need to compare like this:
if (Serial.Available)>16 { // 17 bytes are available
if (serialRead() == "A"){
if (serialRead() == "B"){
if (serialRead() == "~"){
if (serialRead() == "2"){
if (serialRead() == "5"){
if (serialRead() == "5"){
if (serialRead() == "~"){
if (serialRead() == "~"){
char1= serialRead(); // will the data always be three characters? Or is it just two?
char2= serialRead();
char3= serialRead();
if (serialRead() == "~"){
if (serialRead() == "~"){
char4= serialRead();
char5= serialRead();
char6= serialRead();
}}}}}}}}}}} // however many are needed to mate with the {s above - use CTRL-T (autoformat) to make it readable

There's much smarter ways to do this, like concatenating all the reads into a string, and then comparing strings.
I'm too tired to work that out right now tho.

If you keep track of how many characters you have read and matched then you can compare the next read to the next at index in your compare string without buffering reads. If you have many sorted compare strings and you reach a point in one where the next read does not match then you can check the next compare string to the current one up to the last character matched and try to continue matching the read from there, etc.

example, compare strings:

AB110
AB120
AB130
FF001

with input A B 1 3 5

would match AB110 for the 1st 3 characters but not the 4th
check the next compare AB120 for the 1st 3 characters from AB110 == AB1, they match
check the last unmatched read == 3 against the 4th character of AB120 == 2, no match
check the next compare AB130 for the 1st 3 characters from AB120 == AB1, they match
check the last unmatched read == 3 against the 4th character of AB130 == 3, they match
read the next (5th) char == 5 and check against the 5th character pf AB130 == 0, no match
check the next compare FF001 for the 1st 4 characters from AB130 and fail at the 1st character...
return 'no match!'

LOL, I made this to show you can't always expect a match. I had a dictionary routine that would insert the unmatched 'word' on that return but in the OP's app there would be an error to handle or ignore.

Advantage of this is that you should be able to use data in PROGMEM even 1 character at a time to match with. It also allows for non-blocking code using mostly short code sequences depending on your match words list.
Ideally with a -lot- of repetitive letter-number sequences the matching would be done in parts optimized for a particular set of data. I chose the example I did to make as easy explanation.

Back in 82 or 83 I saw a much trickier version of this in a copy of Creative Computing that purported to be able to store a dictionary where every word that started with the same characters is a linked continuation or branch from another word. I screwed with that but it got big quick for a few reasons. The neat thing was being able to match a word being typed in as it was being typed in rather than buffer-and-wait-then-search. I was coding for machines that ran 1 MHz to 4 MHz, 16k to 64k RAM and floppy drives at the time so having the search/match done in a blink after the last key was typed was viewed with surprise back then. What I used was pretty much as outlined above.

I didn't buffer needlessly when I had 16k-64k RAM and I sure don't with only 2k!

PaulS:
In any case, POST YOUR CODE!

Here's the full code. Also see the video afterwards of this code running.
I forgot to get some slow acceleration/deceleration to show the resolution problem.

Info is jumbled together just for testing, I'm waiting on more parts in the mail to expand my display.
Speed is on left (3 digits) then gear (1 digit) then RPM (4 digits)

//X-Sim details

RPM = ~a01~
8 bit resolution
dezimal output

MPH = ~a02~
8 bit resolution
dezimal output

Gear = ~a03~
8 bit resolution
dezimal output


comport speed 115200
Pause of 20ms
R~a01~S~a02~G~a03~
//ARDUINO CODE

/*
rFactor Interface
Kelvyn Panici - 2011

Current X-Sim Limits:
  Tachometer 0-8012RPM
  Speed 0-255MPH
  Gear 1-5, N, R
Current Hardcoded Values:
  RPM = 8012           (See line 59)
  Speed = 255MPH       (See line 132)
  Green Shiftlights = 7650
  Red Shiftlights = 7725
  White Shiftlights = 7825
  */
  
  
#include <TM1638.h>
#define RED TM1638_COLOR_RED
#define GREEN TM1638_COLOR_GREEN

TM1638 display1(22,23,24); //for Arduino Mega
//Clock -> Pin 23 (White)
//DIO -> Pin 22 (Yellow)
//STB0 -> Pin 24 (Green)

//TM1638 display1(2,3,4); //for Arduino Nano
//Clock -> Pin 3 (White)
//DIO -> Pin 2 (Yellow)
//STB0 -> Pin 4 (Green)

char kind_of_data;
boolean slStatusG=0; //Used to trigger shiftlights on and off. They only update once instead of repeatly.
boolean slStatusR=0; //Used to trigger shiftlights on and off.
long previousSLtime = 0;

void setup(){

  pinMode(48, OUTPUT);
  pinMode(49, OUTPUT);
  pinMode(50, OUTPUT);
  pinMode(51, OUTPUT);
  digitalWrite(48, HIGH);
  digitalWrite(49, HIGH);
  digitalWrite(50, HIGH);
  digitalWrite(51, HIGH);
                                  
  
  Serial.begin(115200);
  display1.setupDisplay(true,1);
}


void loop(){
//****************************** READ DATA FROM SERIAL **********************************
  while (Serial.available() > 0){
    kind_of_data = Serial.read();
    if (kind_of_data == 'R' ) Read_Rpm();
    if (kind_of_data == 'S' ) Read_Speed();
    if (kind_of_data == 'G' ) Read_Gear();
  }   
}
//****************************** READ DATA FROM SERIAL END ******************************


//****************************** RPM DISPLAY CONTROL ************************************
void Read_Rpm(){

  int RPM = 0;
  delay(1);
  int Rpm100 = Serial.read();
  delay(1);
  int Rpm10 = Serial.read();     //These three lines read in each digit of the 0-255 RPM value.
  delay(1);
  int Rpm1= Serial.read();

  Rpm100 = ((Rpm100)-48)*100;
  Rpm10 = ((Rpm10)-48)*10;       //AlexOki's conversions.
  Rpm1 = ((Rpm1)-48)*1;

  if (Rpm10 < 0 && Rpm1 < 0){Rpm100 = Rpm100/100;Rpm10 = 0;Rpm1 = 0;}
  if (Rpm1 < 0){Rpm100 = Rpm100/10;Rpm10 = Rpm10/10;Rpm1 = 0;}          //AlexOki's conversions.

  int hisRPM = Rpm100+Rpm10+Rpm1;  //Combines three digits back into a single integer.                      
  RPM=((hisRPM-127)*62.5);      //Converts 0-255 values into 0-8012RPM values.
  
  int digit1 = (RPM/1000);
  int digit2 = ((RPM/100)-(digit1*10));               //Splits RPM integer back into individual digits.
  int digit3 = ((RPM/10)-(digit1*100)-(digit2*10));
  int digit4 = (RPM-(digit1*1000)-(digit2*100)-(digit3*10));
         
  display1.setDisplayDigit(digit1,4,false);
  display1.setDisplayDigit(digit2,5,false);   //Displays digits individually on TM1638 module.
  display1.setDisplayDigit(digit3,6,false);
  display1.setDisplayDigit(digit4,7,false);
    
    
    
//****************************** Shiftlight control ************************************   

  if (RPM>7650){                   //Turns on Green LEDs for revs over 7650.
    if (slStatusG==0){
      display1.setLED(GREEN,0);
      display1.setLED(GREEN,1);
      display1.setLED(GREEN,2);
      display1.setLED(GREEN,3);
    }
    slStatusG = 1;
  }
  else{
    if (slStatusG==1){          //Turns them back off.
      display1.setLED(0,0);
      display1.setLED(0,1);
      display1.setLED(0,2);
      display1.setLED(0,3);
    }
    slStatusG = 0 ;
  }
    if (RPM>7725){          //Turns on Red LEDs for revs over 7825.
      if (slStatusR==0)
        display1.setLED(RED,4);
        display1.setLED(RED,5);
        display1.setLED(RED,6);
        display1.setLED(RED,7);
      slStatusR = 1;
    }
    else{
      if (slStatusR==1)          //Turns them back off.
        display1.setLED(0,4);
        display1.setLED(0,5);
        display1.setLED(0,6);
        display1.setLED(0,7);
      slStatusR = 0 ;
    }
    unsigned long currentSLtime = millis();
    if ((RPM>7825)&&(currentSLtime - previousSLtime > 100)){          //Turns on WHITE LEDs for revs over 7825. They also flash.
      digitalWrite(48, LOW);
      digitalWrite(49, LOW);
      digitalWrite(50, LOW);
      digitalWrite(51, LOW);
      previousSLtime = currentSLtime;
    }
    else{          //Turns them back off.
      digitalWrite(48, HIGH);
      digitalWrite(49, HIGH);
      digitalWrite(50, HIGH);
      digitalWrite(51, HIGH);
    }
}



//****************************** SPEED DISPLAY CONTROL ************************************

void Read_Speed(){
  delay(1);
  int Speed100 = Serial.read();
  delay(1);
  int Speed10 = Serial.read();    //These three lines read in each digit of the 0-255 value.
  delay(1);
  int Speed1= Serial.read();

  Speed100 = ((Speed100)-48)*100;
  Speed10 = ((Speed10)-48)*10;      //AlexOki's conversions.
  Speed1 = ((Speed1)-48)*1;

  if (Speed10 < 0 && Speed1 < 0){Speed100 = Speed100/100;Speed10 = 0;Speed1 = 0;}
  if (Speed1 < 0){Speed100 = Speed100/10;Speed10 = Speed10/10;Speed1 = 0;}        //AlexOki's conversions.

  int hisSpeed = Speed100+Speed10+Speed1;   //Combines three digits back into a single integer.  

  int Speed=((hisSpeed-127)*1.992/1.609344);  //Converts 0-255 values into MPH values.

  int sDigit1 = Speed/100;
  int sDigit2 = Speed/10 - sDigit1*10;          //Splits Speed integer back into individual digits
  int sDigit3 = Speed - sDigit1*100 - sDigit2*10; 
           
  if (sDigit1==0)                //First-digit-zero handling for speedmeter.
    display1.clearDisplayDigit(0, false);// Prevents first digit from sticking.
   else
    display1.setDisplayDigit(sDigit1,0,false);
    
  if ((sDigit1==0)&&(sDigit2==0))                //Second-digit-zero handling for speedmeter.
    display1.clearDisplayDigit(1, false); // Prevents second digit from sticking.
  else
    display1.setDisplayDigit(sDigit2,1,false);
    
  display1.setDisplayDigit(sDigit3,2,false);  //Displays 3rd digit individually on TM1638 module.

}




//****************************** GEAR DISPLAY CONTROL ************************************

void Read_Gear(){

  int Gear = 0;
  delay(1);
  int Gear100 = Serial.read();
  delay(1);
  int Gear10 = Serial.read();        //These three lines read in each digit of the 0-255 Gear value.
  delay(1);
  int Gear1= Serial.read();

  Gear100 = ((Gear100)-48)*100;
  Gear10 = ((Gear10)-48)*10;       //AlexOki's conversions.
  Gear1 = ((Gear1)-48)*1;
  
  if (Gear10 < 0 && Gear1 < 0){Gear100 = Gear100/100;Gear10 = 0;Gear1 = 0;}     //AlexOki's conversions.
  if (Gear < 0){Gear100 = Gear100/10;Gear10 = Gear10/10;Gear1 = 0;}

  Gear = Gear100+Gear10+Gear1;    //Combines three digits back into a single integer.  

  switch (Gear) { //Switch that displays correct gear on TM1638
    case 102:{
      //byte values[] = { 0, 0, 0, 80, 0, 0, 0, 0 };     //Reverse Gear. Working with TM1638 developer to sort out display of this.
      //display1.setDisplay(values);
      //display1.setDisplayDigit(0b01010000,3,false,FONT_DEFAULT);
    }
      break;
    case 127:{                                           //Neutral Gear. Working with TM1638 developer to sort out display of this.
      //byte values[] = { 0, 0, 0, 84, 0, 0, 0, 0 };
      //display1.setDisplay(values); 
      //display1.setDisplayDigit(0b01010100,3,false,FONT_DEFAULT);
    }
      break;
    case 153:                                    //First gear
      display1.setDisplayDigit(1,3,false);
      break;
    case 179:                                    //Second gear
      display1.setDisplayDigit(2,3,false);
      break;
    case 204:                                    //Third gear
      display1.setDisplayDigit(3,3,false);
      break;
    case 230:                                    //Fourth gear
      display1.setDisplayDigit(4,3,false);
      break;
    case 255:                                    //Fifth gear
      display1.setDisplayDigit(5,3,false);
      break;
  }
}
  while (Serial.available() > 0){

kind_of_data = Serial.read();
    if (kind_of_data == 'R' ) Read_Rpm();

...

//****************************** RPM DISPLAY CONTROL ************************************
void Read_Rpm(){

int RPM = 0;
  delay(1);
  int Rpm100 = Serial.read();
  delay(1);
  int Rpm10 = Serial.read();    //These three lines read in each digit of the 0-255 RPM value.
  delay(1);
  int Rpm1= Serial.read();

No. You have established you have a single byte in the serial buffer here:

 while (Serial.available() > 0)

Then you go ahead and read at least 4 bytes. That is almost certainly going to fail. The delays might help a bit but they are a bandaid. Perhaps what GoForSmoke showed - if you are expecting 17 bytes, wait until you get them. Or probably better, just read into a buffer, and then when you get a terminator (eg. a newline) then interpret the buffer.

This subject comes up so often I did a forum post about it here:

Using delay is going to slow the program down for sure. Using the technique I show in the post above doesn't.

In order to get the accuracy I want for RPMs and Speed, i'm now using "16 bit resolution" in X-Sim.

I like that idea. Since I plan to scale up and be transmitting laptimes, engine temperatures, and a few other things, the faster I can get the COM data in, the better.

I've used the code you wrote to display the raw data coming out of X-Sim, however i'm having trouble parsing it for each incoming value (RPM, Speed, Gear).
Do I want to have multiple buffers for each, or somehow split the output?

First gather a "line" of data into a single buffer. Then, by examining that you can work out what it is (eg. RPM, speed, etc.) can't you?

Give us some examples of the data you expect.

Right now i've got the "Datapacket with axisinformations" as in the above picture, set like this.

R~a01~S~a02~G~a03~~10~

Where R, S, and G are characters. (I was using this before to parse the data)

~a01~ is the RPM in "16 bit resolution, dezimal" X-Sim settings. It has values of 0-65535
~a02~ is the Speed in "16 bit resolution, dezimal" X-Sim settings. It has values of 0-65535
~a03~ is the Gear in "8 bit resolution, dezimal" X-Sim settings. It has vales of 0-255.
~10~ is the linefeed.

(I could make the gear also in 16 bit if that makes things easier, but since it will only have 9 unique values at most (R-N-1-2-3-4-5-6-7), that seems like a waste.)

I want to have the values of RPM, speed and gear each in separate variables, which I can then put through formulas to convert to real-world numbers. (and then display.)

Right. So really you are getting:

RxSyGz\n

So you just need to collect (say) up to 10 bytes, and when you get a linefeed analyze the buffer.

The buffer should contain something like this:

buf [0] will be 'R'
buf [1] (x) will be the RPM (0 to 255)
buf [2] will be 'S'
buf [3] (y) will be the speed (0 to 255)
buf [4] will be 'G'
buf [5] (z) will be the gear (0 to 255)
buf [6] will be a linefeed character (ie. the number 10)

Now go ahead and use that data.

Side note, same subject. In your code you have this

if (input_pos < (MAX_INPUT - 1))

why is this better than

if (input_pos <= MAX_INPUT)

Just curious if one is more efficient than the other

But I want to have values of 0-65535 for RPM and speed.
There just isn't enough resolution in 255 steps for an 8000RPM tachometer.

Suppose you know that your serial data will always fit the pattern R#S#G# where # is a variable length number of digits... or that there is a transmission error that may cause you to ignore that line.

You can use a state engine to read those. Each time through loop() you process at most 1 character or process and display your data. Arduino could probably keep up with this at 1 MHz with cycles to spare.

During state 0 when a character is available if you read an R you set state to 1 then return else return.
During state 1 when a character is available if you read an S you set state to 2 then return else use digits as they come in to make your RPM value and return after each. Anything else read, you set state to 0 then return.
During state 2 when a character is available if you read a G you set state to 3 then return else use digits as they come in to make your RPM value and return after each. Anything else read, you set state to 0 then return.
During state 3 you have all 3 values to process and set leds/display. Once done, set state to 0 then return.

Using digit characters to make a value... start with value = 0. For each new digit,
value = value * 10 + (digit-ascii - '0'); // ascii '0' = 48

You might want to check your values for error in case an S or G gets lost in transmission but only need to do so to keep from overflowing value. Thus if a new digit is read and value is already more than (max possible value - digit) / 10, you have an error, set state to 0 and return, LOL.

A state engine is a very simplified version of how I do things naturally. What I do depends of what's been happening. If I boil an egg I get the water to a boiling state first, then remember the time and drop the egg in (next state). I then watch for end time, turn the heat off and take the egg out. I -may- leave the kitchen and return many times during the process but every time I know the state of the operation when I return and act accordingly. Some people put the egg in before the water boils, I don't.

Buffering usually leads to using string.h functions or worse, using String objects. If you process a buffered character array without string.h functions then what do you do that you could not as each character comes in? In this and many other cases, nothing. You don't even identify errors until you process the buffer.

Goofballtech:
Side note, same subject. In your code you have this

if (input_pos < (MAX_INPUT - 1))

why is this better than

if (input_pos <= MAX_INPUT)

Just curious if one is more efficient than the other

They are different things. Say MAX_INPUT is 10. We need to allow room for the final 0x00 (string terminator). Se we can only keep adding while we are less than 9 (because we add one then).

 // keep adding if not full ... allow for terminating null byte
        if (input_pos < (MAX_INPUT - 1))
          input_line [input_pos++] = inByte;

So if we are at position 8, we pass the test, put the byte in and leave input_pos at position 9 (the last valid position, because a 10-byte array goes from 0 to 9). That leaves room for the terminating 0x00.

Your suggested change of "input_pos <= MAX_INPUT" would actually allow you to write to position 10 (which is not valid) and then put the 0x00 in position 11, which is also not valid.

Panici:
But I want to have values of 0-65535 for RPM and speed.
There just isn't enough resolution in 255 steps for an 8000RPM tachometer.

In your original post, on page 1, you checked "8-bit resolution (0 to 255) LowRes" - so you are getting 0 to 255 whether you like it or not. :wink:

If you want more resolution you have to check 16-bit resolution and then you would need to have two bytes for each value (and muck around a bit turning them into an unsigned integer) when you receive them.

You could also consider checking "decimal output" which might be somewhat safer, otherwise if the reading happens to be 10 of something it will look like a newline. If you do that then you need to change the parsing to be a bit different.

I would be tempted to make a state machine like GoForSmoke suggested, use "decimal output" and then just switch states. That is pretty simple to do. :stuck_out_tongue: