Serial Input Basics: example 5, question

This thread gives an outstanding explanation of the use of serial communications:
http://forum.arduino.cc/index.php?topic=396450.0
Robin2 gives in example 5 a means to send in one message various values: integer, float, character,..
I use this with succes to send 3 floats, 2 integers and one character by HC-12 radio.
Example: <99.99,22.25,3.47,11,22,A>

However, from time to time the receiver crashes when receiving messages; using serial monitor debugging I discovered that the receiving controller always crashes while processing the last string (the character, A).

To debug, I inserted serial output prints after each parsing command:

  strtokIndx = strtok(tempChars, ",");     // get the first part - a float
  integer1FromPC = atof(strtokIndx);     // convert this part (first value) to a float: dutycycle
  Serial.print(F("   1 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer2FromPC = atof(strtokIndx);     // convert this part (second value) to a float: temperature
  Serial.print(F(" 2 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer3FromPC = atof(strtokIndx);     // convert this part (third value) to a float: battery voltage
  Serial.print(F(" 3 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer4FromPC = atoi(strtokIndx);     // convert this part (fourth value) to an integer: minutes past the hour
  Serial.print(F(" 4 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer5FromPC = atoi(strtokIndx);     // convert this part (fifth value) to an integer: hour of the day
  Serial.print(F(" 5 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();

When a crash occurs, I still get the value "5" on the serial monitor, but not the value "6", always.

This is the same parsing code as used in Robin2 example 5.

I suspect that once in a while the message is corrupted during transmission and that either a null character or something else that is not right is being received.

Would it be possible to include code that distinguishes, in the message to be parsed, whether there are either 0 or null values, or some other way to prevent the code execution to cause the controller to crash?

how have you declared messageFromPC ? as a char * i suspect but what size ? (the terminating \0 needs to be part of the string)

As usual, it would be useful to provide the ENTIRE code instead of just a snippet.

Deva_Rishi:
how have you declared messageFromPC ? as a char * i suspect but what size ? (the terminating \0 needs to be part of the string)

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing
// variables to hold the parsed data
char messageFromPC[numChars] = {0};
float integer1FromPC;  // value1 (dutycyclevalue=soil moisture level)
float integer2FromPC;  // value2 (temperature at soil moisture sensor)
float integer3FromPC;  // battery voltage
int integer4FromPC;  // integer minutes (transmission time-instance, minutes past the hour)
int integer5FromPC;  // integer hour (transmission time-instance, hour of the day)

32 bytes, actually taken from Robin2's example. Would that be correct in this context?

vaj4088:
As usual, it would be useful to provide the ENTIRE code instead of just a snippet.

I would gladly do that but it is 1035 lines of code; and I think the issue is with a very specific part, in the parsing section: so with all due respect I hope to spare you the nightmare of wading through what might be avoided maybe by starting with the part that starts the crash and MCU restart?

You may have avoided one nightmare but created another by forcing unpaid helpers to play guessing games and ask questions about unseen portions of the code.

Do you REALLY want your help delayed?

vaj4088:
You may have avoided one nightmare but created another by forcing unpaid helpers to play guessing games and ask questions about unseen portions of the code.

Do you REALLY want your help delayed?

Here you go, I hope this helps; thanks in advance.

strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();

When parsing the last piece, strtok is looking for a ',' which does not exist. In testing, it has not appeared to make a difference between using strtok(NULL, NULL) or strtok(NULL, ","), but perhaps there is something in your case which does make a difference.

There are two thing to test.

First, see if strtok(NULL,NULL) improves things.

Second, try formulating the transmitted string like this with a period after the letter
<99.99,22.25,3.47,11,22,A.>

Then parse with

strtok(NULL, ".")

cattledog:

strtokIndx = strtok(NULL, ","); // this continues where the previous call left off

strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
 // sensor identification, sensor "A" or sensor "B", or C or D
 Serial.println(F(" 6 "));
 Serial.println();




When parsing the last piece, strtok is looking for a ',' which does not exist. In testing, it has not appeared to make a difference between using strtok(NULL, NULL) or strtok(NULL, ","), but perhaps there is something in your case which does make a difference.

There are two thing to test.

First, see if strtok(NULL,NULL) improves things.

Second, try formulating the transmitted string like this with a period after the letter
<99.99,22.25,3.47,11,22,A.>

Then parse with

strtok(NULL, ".")

Great, thanks a lot for that, I had not noticed this.
Just a question here: would/could there be a reason why most of the received messages do get parsed succesfully?
Then, your suggestion is to include a . after the A however should that not be a , ?

For sure I will be testing both suggestions and let you know asap.

EDIT: sorry, I misread your suggestion "..parse with strtok(NULL, "."); evidently this requires A.

brice3010:
Here you go, I hope this helps; thanks in advance.

Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.

Just a question here: would/could there be a reason why most of the received messages do get parsed succesfully?

My guess would be that in the rare, failing case, there is no null terminator.

If there is no null terminator on receivedChars[] or tempChars[] I believe that the behavior of strtok will be undefined when parsing the last piece. The esp8266 also has a pretty short watchdog timer which causes resets if things get blocked

Why there would be a missing '\0' is a mystery.

Robin2:
Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.

Good idea, will be done asap. Results and program will be posted.

cattledog:
My guess would be that in the rare, failing case, there is no null terminator.

If there is no null terminator on receivedChars[] or tempChars[] I believe that the behavior of strtok will be undefined when parsing the last piece. The esp8266 also has a pretty short watchdog timer which causes resets if things get blocked

Why there would be a missing '\0' is a mystery.

The strtokIndx = strtok(NULL, NULL); resulted in continual crashes, no parsing possible. Next will be the A. test as well as a bare-bones program version (Robin2).

The strtokIndx = strtok(NULL, NULL); resulted in continual crashes, no parsing possible.

Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged. If that is the situation for this test, then there must be something wrong with the null termination.

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off
 //strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();

Actually, I have higher hopes for the addition of another token after the A. It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.

cattledog:
Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged. If that is the situation for this test, then there must be something wrong with the null termination.

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off

//strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();




Actually, I have higher hopes for the addition of another token after the A. It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.

Indeed, only the last use of strtok had been changed.
Next today the A. test and corresponding strtok change will be done.

Would it not be much more elegant to simply add a ',' as a terminating character here

if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars-1) {
                    ndx = numChars - 2;             // and forcefully leave 1 extra character of space here
                }
            }
            else {
                receivedChars[ndx] = ',';          // put an extra separator
                receivedChars[ndx+1] = '\0';   // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }

Robin2:
Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.

Sortened bare-bones version included.

To clarify, here is the code used by the transmitter in the TX code section:

    Serial.print('<'); // this section for HC-12 transmission
    Serial.print(cycleValue);  // ref Robin2's Serial Output Basics
    Serial.print(',');   // http://forum.arduino.cc/index.php?topic=396450.0
    //    Serial.print(temperatureT1); // send temperature read from analog input temperaturePin
    Serial.print(get3231Temp()); // read DS3231 temperature
    Serial.print(',');
    Serial.print(batteryVoltage); // send battery voltage
    Serial.print(',');
    Serial.print(now.minute(), DEC); // send minute
    Serial.print(',');
    Serial.print(now.hour(), DEC); // send hour
    Serial.print(",A>"); // send sensor identifier
    Serial.println();

The pending test (ref cattledog) with use of A. is not yet shown here.

ESP8266_parse_test.ino (6.98 KB)

Deva_Rishi:
Would it not be much more elegant to simply add a ',' as a terminating character here

if (recvInProgress == true) {

if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars-1) {
                    ndx = numChars - 2;            // and forcefully leave 1 extra character of space here
                }
            }
            else {
                receivedChars[ndx] = ',';          // put an extra separator
                receivedChars[ndx+1] = '\0';  // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }

The issue with that I think is that , are used as separator, and termination is done with endmarker: this allows the program to differentiate between "continuous" data and "last" data.
What advantage would the use of , as endmarker have?

What advantage would the use of , as endmarker have?

what was suggested earlier just so strtok() is looking for something that it can find.

cattledog:
Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged. If that is the situation for this test, then there must be something wrong with the null termination.

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off

Actually, I have higher hopes for the addition of another token after the A. It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.

my point also, when a '>' endmarker is received do not just terminate with a NULL but add a ',' separator as well. I can't see anything being wrong with the terminating NULL other then strtok() not finding it !

Deva_Rishi:
what was suggested earlier just so strtok() is looking for something that it can find.
my point also, when a '>' endmarker is received do not just terminate with a NULL but add a ',' separator as well. I can't see anything being wrong with the terminating NULL other then strtok() not finding it !

Indeed, just to recap: now A is being sent without anything behind it, so placing a . or a , after the A and have strtok identify this character: strtokIndx = strtok(NULL, "."); in case of A.
Right?