[SOLVED] Need help with using marker, parsing data, and clearing serial buffer

Hello, I need help on my project to implement data markers, parsing, and clearing any leftover data on the serial buffer every single loop. I intend to use this program as a receiver and controller for a ground drone.

Here is a copy of my code:

// Receive with start- and end-markers combined with parsing

const byte maxDataLength = 12;
char receivedData[maxDataLength];
char tempData[maxDataLength];        // temporary array for use by strtok() function

// variables to hold the parsed data
int pin = '\0';
int value = 0;

boolean newData = false;

int pins [] = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13};
int pinsize = 11;

//============

void setup() {
  for(int a=0; a<pinsize; a++){
    pinMode(pins[a], OUTPUT);
  }
  Serial.begin(9600);
  //Serial.setTimeout(20);
  Serial.print(F("This program expects 2 pieces of data, the pin number, "));
  Serial.print(F("and value (0 to 255 for PWM, 0 and positive integer for digital"));
  Serial.println(F(" enter data in this style <pin, value>. PWM Pins are 3, 5, 6, 9, 10, 11"));
  Serial.println(F("Ready for command"));
  Serial.println();
}

//============

void loop() {
  int p;
  checkData();
  if (newData == true) {
    strcpy(tempData, receivedData); //copy the recievedChars to tempChars
    // this temporary copy is necessary to protect the original data
    //   because strtok() replaces the commas with \0
    parseData();
    showParsedData();
    p=pin-2;
    if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){
      if (value != 0){
        digitalWrite(pins[p], HIGH);
      }
      else{
        digitalWrite(pins[p], LOW);
      }
    }
    else{
      analogWrite(pins[p], value);
    }
    newData = false;
  }
  int a = Serial.read();
  Serial.flush();
  a = '\0';
}

//============

void checkData() {
  static boolean receiving = false;
  static byte n = 0;
  char startMarker = '<';
  char endMarker = '>';
  char c;

  while (Serial.available() > 0 && newData == false) {
    c = Serial.read();

    if (receiving == true) { // this one is the second
      if (c != endMarker) {
        receivedData[n] = c;
        n++;
        if (n >= maxDataLength) {
          n = maxDataLength - 1;
        }
      }
      else { //this is the last, notice the while loop loser on the newData=true
        receivedData[n] = '\0'; //clear the data that is now unused
        receiving = false;
        n = 0;
        newData = true;
      }
    }

    else if (c == startMarker) { //this one gets executed first, recvInProgress = true, and then the while do the loop, resulting the if(recvInProgress -- true) started
      receiving = true;
    }
  }
}

//============

void parseData() {

  // split the data into its parts
  char * indx; // this is used by strtok() as an index

  indx = strtok(tempData, ",");     // get the first part - the pin number
  pin = atoi(indx);
  
  indx = strtok(NULL, ","); // this continues where the previous call left off
  value = atoi(indx);     // convert this part to an integer


}

//============

void showParsedData() {
  Serial.print("Pin = ");
  Serial.println(pin);
  Serial.print("Value = ");
  Serial.println(value);
}
// http://forum.arduino.cc/index.php?topic=288234.60 , page 5
// credits to Robin2 http://forum.arduino.cc/index.php?action=profile;u=174714, for the tutorial about parsing and using marker, also the piece of code

Which is an "upgrade" from this code:

// Receive with start- and end-markers combined with parsing

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use by strtok() function

// variables to hold the parsed data
int pin = '\0';
int value = 0;

boolean newData = false;

int pins []={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13};
int pinsize= 11;

//============

void setup() {
  for(int a=0; a<pinsize; a++){
    pinMode(pins[a], OUTPUT);
  }
  Serial.begin(9600);
  Serial.println("This operation expects 2 pieces of data, the pin number (PWM), and value (0 to 255)");
  Serial.println("Enter data in this style <pin, value>. Only PWM Pins, which is pin 3, 5, 6, 9, 10, 11");
  Serial.println();
}

//============

void loop() {
  int p;
  recvWithStartEndMarkers();
  if (newData == true) {
    strcpy(tempChars, receivedChars); //copy the recievedChars to tempChars
    // this temporary copy is necessary to protect the original data
    //   because strtok() replaces the commas with \0
    parseData();
    showParsedData();
    p=pin-2;
    analogWrite(pins[p], value);
    newData = false;
  }
}

//============

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '<';
  char endMarker = '>';
  char rc;

  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (recvInProgress == true) { // this one is the second
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else { //this is the last, notice the while loop loser on the newData=true
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }

    else if (rc == startMarker) { //this one gets executed first, recvInProgress = true, and then the while do the loop, resulting the if(recvInProgress -- true) started
      recvInProgress = true;
    }
  }
}

//============

void parseData() {

  // split the data into its parts
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the pin number
  pin = atoi(strtokIndx);
  
  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  value = atoi(strtokIndx);     // convert this part to an integer


}

//============

void showParsedData() {
  Serial.print("Pin = ");
  Serial.println(pin);
  Serial.print("Value = ");
  Serial.println(value);
}
// http://forum.arduino.cc/index.php?topic=288234.60 , page 5
// credits to Robin2 http://forum.arduino.cc/index.php?action=profile;u=174714, for the tutorial about parsing and using marker, also the piece of code

Sorry a long one :confused:

The main point of "upgrading" the code is to extend the control so the digital pins can be used and adding reliability to the serial buffer so it automatically dump every junk/garbled data that is received, so the arduino can stand by for a longer time.

However, there are problems in the "improved" code, that the data is not sent back to the serial monitor, or sent back incorrectly (which means that the program did not do as intended), as the picture in attachment
Picture 1: the data sent is <2, 1>
Picture 2: the data sent is <3, 255> (no reply), <2, 1> (first reply), and <2,1> (second reply). Further sending <2,1>, <3,255> results in no replies).

There is no compiler error, and the LEDs I used for testing did not light up either (obviously).

Will post the build cabling later as I do not have fritzing installed yet.

Any help is greatly appreciated!! :slight_smile:

Your pictures are:

don't you think just copy/paste of the serial console would be easier :slight_smile:

in your code you have

int pins [] = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13};
int pinsize = 11;

//============

void setup() {
  for (int a = 0; a < pinsize; a++) {
    pinMode(pins[a], OUTPUT);
  }
....

are you sure you have 11 entries in your array? best is to let the compiler work for you and doint pinsize = sizeof(pins)/sizeof(pins[0]);

I'm not sure how you get the data into your program? typing from the serial monitor?

Leviathan616:
Which is an "upgrade" from this code:

You seem to have made irrelevant changes to the function recvWithStartEndMarkers() which means I am not going to wast time studying your code.

Leave unchanged the parts that don't need to be changed and then we can all concentrate our attention on the necessary changes, Or, put another way, If it works, don't fix it

When it is all working you can prettify any parts that take your fancy.

...R

Still no luck... (pic3)

First entry is <2,1>, no reply
Second entry is <3,255>, replied by 325 and 0
Third entry is <3,0>, no reply
Fourth entry is <3,0>, replied by 30 and 0
Fifth entry is <3,5>, no reply

Total 5 entries with 2 replies

Maybe the parsing error? Because the data seems to be in an incorrect place, but no reply is still a concern

I don't know how to post images directly into the main reply, can you teach me?
p.s. I want to paste the images so that visual-learners (like me) can learn faster without the need to imagining things, hope you wouldn't mind :slight_smile: :slight_smile:

Thanks!

Robin2:
You seem to have made irrelevant changes to the function recvWithStartEndMarkers() which means I am not going to wast time studying your code.

here is a diff of the codes, it's just cosmetic

Leviathan616:
Still no luck... (pic3)

First entry is <2,1>, no reply
Second entry is <3,255>, replied by 325 and 0
Third entry is <3,0>, no reply
Fourth entry is <3,0>, replied by 30 and 0
Fifth entry is <3,5>, no reply

Total 5 entries with 2 replies

Maybe the parsing error? Because the data seems to be in an incorrect place, but no reply is still a concern

I don't know how to post images directly into the main reply, can you teach me?
p.s. I want to paste the images so that visual-learners (like me) can learn faster without the need to imagining things, hope you wouldn't mind :slight_smile: :slight_smile:

Thanks!

why do you have this (stupid) idea to remove data from the Serial at the end of the loop.

 int a = Serial.read();
  Serial.flush();
  a = '\0';

when you don't even know what's there ?

for posting images, just read how to use the forum (even if images are a bit complicated) - you could also read this for example

Reverting back the parts from Robin2's code:

// Receive with start- and end-markers combined with parsing

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use by strtok() function

// variables to hold the parsed data
int pin = '\0';
int value = 0;

boolean newData = false;

int pins [] = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13};
int pinsize = sizeof(pins);

//============

void setup() {
  for(int a=0; a<pinsize; a++){
    pinMode(pins[a], OUTPUT);
  }
  Serial.begin(9600);
  //Serial.setTimeout(20);
  Serial.print(F("This program expects 2 pieces of data, the pin number, "));
  Serial.print(F("and value (0 to 255 for PWM, 0 and positive integer for digital"));
  Serial.println(F(" enter data in this style <pin, value>. PWM Pins are 3, 5, 6, 9, 10, 11"));
  Serial.println(F("Ready for command"));
  Serial.println();
}

//============

void loop() {
  int p;
  recvWithStartEndMarkers();
  if (newData == true) {
    strcpy(tempChars, receivedChars); //copy the recievedChars to tempChars
    // this temporary copy is necessary to protect the original data
    //   because strtok() replaces the commas with \0
    parseData();
    showParsedData();
    p=pin-2;
    if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){
      if (value != 0){
        digitalWrite(pins[p], HIGH);
      }
      else{
        digitalWrite(pins[p], LOW);
      }
    }
    else{
      analogWrite(pins[p], value);
    }
    newData = false;
  }
  int a = Serial.read();
  Serial.flush();
  a = '\0';
}

//============

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '<';
  char endMarker = '>';
  char rc;

  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (recvInProgress == true) { // this one is the second
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else { //this is the last, notice the while loop loser on the newData=true
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }

    else if (rc == startMarker) { //this one gets executed first, recvInProgress = true, and then the while do the loop, resulting the if(recvInProgress -- true) started
      recvInProgress = true;
    }
  }
}

//============

void parseData() {

  // split the data into its parts
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the pin number
  pin = atoi(strtokIndx);
  
  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  value = atoi(strtokIndx);     // convert this part to an integer


}

//============

void showParsedData() {
  Serial.print("Pin = ");
  Serial.println(pin);
  Serial.print("Value = ");
  Serial.println(value);
}
// http://forum.arduino.cc/index.php?topic=288234.60 , page 5
// credits to Robin2 http://forum.arduino.cc/index.php?action=profile;u=174714, for the tutorial about parsing and using marker, also the piece of code

Still same problem:
1st entry: <2,1>, no reply
2nd entry: <2,0>, replied by 21 (pin) and 0 (value)
3rd entry: <3,255>, no reply
4th entry: <3,255>, replied by 5 and 0
5th entry: <3,255>, no reply
6th entry: <3,255>, replied by 3 and 25, LED green on
7th entry: <3,0>, replied by 0 and 0
8th entry: <3,0>, replied by 0 and 0
9th entry: <3,0>, replied by 0 and 0

Total 9 entries, 6 replies

p.s. Sorry Robin2, but the reason I changed the code is to understand more about the parsing. At first, I think that the variables are "special" variables, which means that you cannot change it to something else (I do not have a good programming background, and maybe screwing something at the first time I tried to change the variables in the code). I also planned to make 3 or more variables in a single datastream as I want to send accelerometer along with any other sensor data from the my land drone to receiver arduino using HC-12 radio comms, and the receiver to change the data for a more human-readable format.

@J-M-L, I assume that the arduino program executes command from top-to-bottom, so I think after the serial data is already processed, all leftovers can be dumped so the buffer is clear. Also, the "upgrade" I mean is this:

if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){
      if (value != 0){
        digitalWrite(pins[p], HIGH);
      }
      else{
        digitalWrite(pins[p], LOW);
      }
    }
    else{
      analogWrite(pins[p], value);
    }

and experimenting on this:

int a = Serial.read();
  Serial.flush();
  a = '\0';

The cosmetic changes is not what I meant the "upgrade", I never have any attention to call that an "upgrade", those are just other minor-trial-and-error-but-expected-to-work experiments for me to understand the code more.

Sorry all for my misleading statement, and thanks for reply!

@J-M-L, I assume that the arduino program executes command from top-to-bottom, so I think after the serial data is already processed, all leftovers can be dumped so the buffer is clear.

flush() does not do anything to the input buffer

do you understand that the "magic" is done in recvWithStartEndMarkers(); but that this function is non blocking. it reads the Serial line, starts accumulating data in the buffer after finding the startMarker but is asynchronous. so it might take many loops to fill in the buffer

What do you think happens if a byte becomes available after recvWithStartEndMarkers(); has returned (it emptied the buffer) but before you actually received the endMarker...

never try to second guess timing on an asynchronous line.

you don't need to worry about crappy stuff staying the input buffer the next call will wait for a startMarker so the looping will get rid of the unnecessary input

J-M-L:
flush() does not do anything to the input buffer

Yes, but I want to clear the output buffer too

J-M-L:
do you understand that the "magic" is done in recvWithStartEndMarkers(); but that this function is non blocking. it reads the Serial line, starts accumulating data in the buffer after finding the startMarker but is asynchronous. so it might take many loops to fill in the buffer

What do you think happens if a byte becomes available after recvWithStartEndMarkers(); has returned (it emptied the buffer) but before you actually received the endMarker...

never try to second guess timing on an asynchronous line.

you don't need to worry about crappy stuff staying the input buffer the next call will wait for a startMarker so the looping will get rid of the unnecessary input

My first thought is that arduino cannot run two things at once, and I thought that the recvWithStartEndMarkers() void will be executed for many loops before going to the next step in void loop(). (Again, I think codes in a top-down sequence single core processor blocks entire process, which is why I don't understand millis() until now, still confused about non-blocking codes)

And as I remove

int a = Serial.read();
  Serial.flush();
  a = '\0';

The serial responds 100% correct:

Pin = 2
Value = 1
Pin = 2
Value = 255
Pin = 2
Value = 0
Pin = 3
Value = 1
Pin = 3
Value = 255
Pin = 3
Value = 0

but the LED is not (analogWrite with value 1 is as bright as value 255, when with the old code the LED is significantly dimmer), other than that, pin 2 is working as intended (as a digital sample).

Thanks! But, can you take a look at this?

if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){
      if (value != 0){
        digitalWrite(pins[p], HIGH);
      }
      else{
        digitalWrite(pins[p], LOW);
      }
    }
    else{
      analogWrite(pins[p], value);
    }

Those lines may be the culprit. The p variable is the arduino uno's PWM pin minus 2 (because the array for pins starts from 2 to 13), so that if sequence is intended to be used for non-PWM pins, while the else is for PWM pin.
But, I don't know what's wrong with that code.

the arduino indeed cannot execute two things at once... so it reads first as many characters as possible and when the buffer is empty go do what is next in the code, top down as you put it... and then the loop loops.

if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){are you sure you want to do ORs there, may be you want to do ANDs :slight_smile:

Note as well the if you do an analogWrite() on a non PWM pin, the function has a special case for that and will set the pin to LOW if the value is less than 128 and HIGH if the value is above

 			case NOT_ON_TIMER:
			default:
				if (val < 128) {
					digitalWrite(pin, LOW);
				} else {
					digitalWrite(pin, HIGH);
				}

J-M-L:
the arduino indeed cannot execute two things at once... so it reads first as many characters as possible and when the buffer is empty go do what is next in the code, top down as you put it... and then the loop loops.

So the void loop() is executed first, then the first void called in the loop, and after that void finished one loop (task still not finished, need more than one loops), the system calls for the void loop() again?

J-M-L:
if ((p != 1)||(p != 3)||(p != 4)||(p != 7)||(p != 8)||(p != 9)){are you sure you want to do ORs there, may be you want to do ANDs :slight_smile:

Big thanks! That should be an AND there, not OR

J-M-L:
Note as well the if you do an analogWrite() on a non PWM pin, the function has a special case for that and will set the pin to LOW if the value is less than 128 and HIGH if the value is above

 			case NOT_ON_TIMER:
		default:
			if (val < 128) {
				digitalWrite(pin, LOW);
			} else {
				digitalWrite(pin, HIGH);
			}

Thanks, I haven't read that one

And one thing:

J-M-L:
you don't need to worry about crappy stuff staying the input buffer the next call will wait for a startMarker so the looping will get rid of the unnecessary input

So the serial buffer is cleared if there is a broken datastream? (Like somehow my data is torn and the start or end marker lost, that data will be dumped and the buffer cleared?)

Also, considering I'm using HC-12 at mode FU3 (Highest power, 9600bps, 15,000bps in the air (I don't know what is baud rate in the air, actually, so I stick with the 9600bps for calculation)

Serial.setTimeout(5);

How fast the timeout can be without breaking my data? Thanks!

Successful code (with serial monitor, not HC-12)

// Receive with start- and end-markers combined with parsing

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use by strtok() function

// variables to hold the parsed data
int pin = '\0';
int value = 0;

boolean newData = false;

int pins [] = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13};
int pinsize = sizeof(pins);

//============

void setup() {
  for(int a=0; a<pinsize; a++){
    pinMode(pins[a], OUTPUT);
  }
  Serial.begin(9600);
  Serial.setTimeout(5);
  Serial.print(F("This program expects 2 pieces of data, the pin number, "));
  Serial.print(F("and value (0 to 255 for PWM, 0 and positive integer for digital"));
  Serial.println(F(" enter data in this style <pin, value>. PWM Pins are 3, 5, 6, 9, 10, 11"));
  Serial.println(F("Ready for command"));
  Serial.println();
}

//============

void loop() {
  int p;
  recvWithStartEndMarkers();
  if (newData == true) {
    strcpy(tempChars, receivedChars); //copy the recievedChars to tempChars
    // this temporary copy is necessary to protect the original data
    //   because strtok() replaces the commas with \0
    parseData();
    showParsedData();
    p=pin-2;
    if ((p != 1)&&(p != 3)&&(p != 4)&&(p != 7)&&(p != 8)&&(p != 9)){
      if (value != 0){
        digitalWrite(pins[p], HIGH);
      }
      else{
        digitalWrite(pins[p], LOW);
      }
    }
    else{
      analogWrite(pins[p], value);
    }
    newData = false;
  }
}

//============

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '<';
  char endMarker = '>';
  char rc;

  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (recvInProgress == true) { // this one is the second
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else { //this is the last, notice the while loop loser on the newData=true
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }

    else if (rc == startMarker) { //this one gets executed first, recvInProgress = true, and then the while do the loop, resulting the if(recvInProgress -- true) started
      recvInProgress = true;
    }
  }
}

//============

void parseData() {

  // split the data into its parts
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the pin number
  pin = atoi(strtokIndx);
  
  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  value = atoi(strtokIndx);     // convert this part to an integer


}

//============

void showParsedData() {
  Serial.print("Pin = ");
  Serial.println(pin);
  Serial.print("Value = ");
  Serial.println(value);
}
// http://forum.arduino.cc/index.php?topic=288234.60 , page 5
// credits to Robin2 http://forum.arduino.cc/index.php?action=profile;u=174714, for the tutorial about parsing and using marker, also the piece of code
// and credits to J-M-L https://forum.arduino.cc/index.php?action=profile;u=438300, discussion topic https://forum.arduino.cc/index.php?topic=558096.msg3805943#msg3805943

Leviathan616:
So the void loop() is executed first, then the first void called in the loop, and after that void finished one loop (task still not finished, need more than one loops), the system calls for the void loop() again?

a void does not mean anything. You are calling functions, returning nothing (void)

I don't know what's your level in C or C++ but the IDE hides for you a main() function which is the entry point of the real program. When you compile a sketch, a main() function is created. The init() and setup() functions are called only once, and then loop() is called in an endless way. So this main() function looks like this (a bit simplified)

int main(void)
{
	init();
	setup();
	for (;;) loop();
	return 0;
}

Big thanks! That should be an AND there, not OR

:slight_smile:

So the serial buffer is cleared if there is a broken datastream? (Like somehow my data is torn and the start or end marker lost, that data will be dumped and the buffer cleared?)

No your function that is expecting a well formed entry starting with a < and ending with a > will get rid of any data until it gets the <, then will accumulate whatever arrives until you receive the >. It's up to you to handle what you receive and decide if it's garbage or not. You need to improve the parser.

Also, considering I'm using HC-12 at mode FU3 (Highest power, 9600bps, 15,000bps in the air (I don't know what is baud rate in the air, actually, so I stick with the 9600bps for calculation)

Serial.setTimeout(5);

How fast the timeout can be without breaking my data? Thanks!

as discussed above, never try to second guess the timing of our serial line. You don't need to use the crappy convenience functions provided with the IDE, they won't do any good to you. Just read stuff when it comes in and analyse what you get.

J-M-L:
here is a diff of the codes, it's just cosmetic

I know.

That's why it is completely pointless. You (or anyone else) should not have to go to the trouble of making a comparison.

...R