Arduino Forum

Using Arduino => Programming Questions => Topic started by: brice3010 on Jan 14, 2019, 08:23 pm

Title: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 14, 2019, 08:23 pm
This thread gives an outstanding explanation of the use of serial communications:
http://forum.arduino.cc/index.php?topic=396450.0 (http://forum.arduino.cc/index.php?topic=396450.0)
Robin2 gives in example 5 a means to send in one message various values: integer, float, character,..
I use this with succes to send 3 floats, 2 integers and one character by HC-12 radio.
Example: <99.99,22.25,3.47,11,22,A>

However, from time to time the receiver crashes when receiving messages; using serial monitor debugging I discovered that the receiving controller always crashes while processing the last string (the character, A).

To debug, I inserted serial output prints after each parsing command:
Code: [Select]

  strtokIndx = strtok(tempChars, ",");     // get the first part - a float
  integer1FromPC = atof(strtokIndx);     // convert this part (first value) to a float: dutycycle
  Serial.print(F("   1 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer2FromPC = atof(strtokIndx);     // convert this part (second value) to a float: temperature
  Serial.print(F(" 2 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer3FromPC = atof(strtokIndx);     // convert this part (third value) to a float: battery voltage
  Serial.print(F(" 3 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer4FromPC = atoi(strtokIndx);     // convert this part (fourth value) to an integer: minutes past the hour
  Serial.print(F(" 4 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  integer5FromPC = atoi(strtokIndx);     // convert this part (fifth value) to an integer: hour of the day
  Serial.print(F(" 5 "));

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


When a crash occurs, I still get the value "5" on the serial monitor, but not the value "6", always.

This is the same parsing code as used in Robin2 example 5.

I suspect that once in a while the message is corrupted during transmission and that either a null character or something else that is not right is being received.

Would it be possible to include code that distinguishes, in the message to be parsed, whether there are either 0 or null values, or some other way to prevent the code execution to cause the controller to crash?




Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 14, 2019, 09:35 pm
how have you declared
Code: [Select]
messageFromPC ? as a char * i suspect but what size ? (the terminating \0 needs to be part of the string)
Title: Re: Serial Input Basics: example 5, question
Post by: vaj4088 on Jan 14, 2019, 09:41 pm
As usual, it would be useful to provide the ENTIRE code instead of just a snippet.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 14, 2019, 09:45 pm
how have you declared
Code: [Select]
messageFromPC ? as a char * i suspect but what size ? (the terminating \0 needs to be part of the string)
Code: [Select]

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing
// variables to hold the parsed data
char messageFromPC[numChars] = {0};
float integer1FromPC;  // value1 (dutycyclevalue=soil moisture level)
float integer2FromPC;  // value2 (temperature at soil moisture sensor)
float integer3FromPC;  // battery voltage
int integer4FromPC;  // integer minutes (transmission time-instance, minutes past the hour)
int integer5FromPC;  // integer hour (transmission time-instance, hour of the day)


32 bytes, actually taken from Robin2's example. Would that be correct in this context?

Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 14, 2019, 09:49 pm
As usual, it would be useful to provide the ENTIRE code instead of just a snippet.

I would gladly do that but it is 1035 lines of code; and I think the issue is with a very specific part, in the parsing section: so with all due respect I hope to spare you the nightmare of wading through what might be avoided maybe by starting with the part that starts the crash and MCU restart?
Title: Re: Serial Input Basics: example 5, question
Post by: vaj4088 on Jan 14, 2019, 09:57 pm
You may have avoided one nightmare but created another by forcing unpaid helpers to play guessing games and ask questions about unseen portions of the code.

Do you REALLY want your help delayed?

Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 14, 2019, 10:18 pm
You may have avoided one nightmare but created another by forcing unpaid helpers to play guessing games and ask questions about unseen portions of the code.

Do you REALLY want your help delayed?


Here you go, I hope this helps; thanks in advance.
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 14, 2019, 10:49 pm
Code: [Select]
strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


When parsing the last piece, strtok is looking for a ',' which does not exist. In testing, it has not appeared to make a difference between using strtok(NULL, NULL) or strtok(NULL, ","), but perhaps there is something in your case which does make a difference.

There are two thing to test.

First, see if strtok(NULL,NULL) improves things.

Second, try formulating the transmitted string like this with a period after the letter
<99.99,22.25,3.47,11,22,A.>

Then parse with

strtok(NULL, ".")


Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 14, 2019, 10:56 pm
Code: [Select]
strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


When parsing the last piece, strtok is looking for a ',' which does not exist. In testing, it has not appeared to make a difference between using strtok(NULL, NULL) or strtok(NULL, ","), but perhaps there is something in your case which does make a difference.

There are two thing to test.

First, see if strtok(NULL,NULL) improves things.

Second, try formulating the transmitted string like this with a period after the letter
<99.99,22.25,3.47,11,22,A.>

Then parse with

strtok(NULL, ".")



Great, thanks a lot for that, I had not noticed this.
Just a question here: would/could there be a reason why most of the received messages do get parsed succesfully?
Then, your suggestion is to include a . after the A however should that not be a , ?

For sure I will be testing both suggestions and let you know asap.

EDIT: sorry, I misread your suggestion "..parse with strtok(NULL, "."); evidently this requires A.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 14, 2019, 11:19 pm
Here you go, I hope this helps; thanks in advance.
Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 14, 2019, 11:30 pm
Quote
Just a question here: would/could there be a reason why most of the received messages do get parsed succesfully?
My guess would be that in the rare, failing case, there is no null terminator.

If there is no null terminator on receivedChars[] or tempChars[] I believe that the behavior of strtok will be undefined when parsing the last piece. The esp8266 also has a pretty short watchdog timer which causes resets if things get blocked

Why there would be a missing '\0' is a mystery.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 06:44 am
Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.
Good idea, will be done asap. Results and program will be posted.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 06:47 am
My guess would be that in the rare, failing case, there is no null terminator.

If there is no null terminator on receivedChars[] or tempChars[] I believe that the behavior of strtok will be undefined when parsing the last piece. The esp8266 also has a pretty short watchdog timer which causes resets if things get blocked

Why there would be a missing '\0' is a mystery.
The strtokIndx = strtok(NULL, NULL); resulted in continual crashes, no parsing possible. Next will be the A. test as well as a bare-bones program version (Robin2).
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 15, 2019, 07:01 am
Quote
The strtokIndx = strtok(NULL, NULL); resulted in continual crashes, no parsing possible.
Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged.  If that is the situation for this test, then there must be something wrong with the null termination.

Code: [Select]

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off
 //strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


Actually, I have higher hopes for the addition of another token after the A.  It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 07:31 am
Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged.  If that is the situation for this test, then there must be something wrong with the null termination.

Code: [Select]

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off
 //strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


Actually, I have higher hopes for the addition of another token after the A.  It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.
Indeed, only the last use of strtok had been changed.
Next today the A. test and corresponding strtok change will be done.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 15, 2019, 10:34 am
Would it not be much more elegant to simply add a ',' as a terminating character here
Code: [Select]
if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars-1) {
                    ndx = numChars - 2;             // and forcefully leave 1 extra character of space here
                }
            }
            else {
                receivedChars[ndx] = ',';          // put an extra separator
                receivedChars[ndx+1] = '\0';   // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 01:53 pm
Can you make a short program that illustrates the problem? I suspect it would help you as well as us.

...R
PS ... I have not tested my examples on an ESP8266.
Sortened bare-bones version included.

To clarify, here is the code used by the transmitter in the TX code section:
Code: [Select]

    Serial.print('<'); // this section for HC-12 transmission
    Serial.print(cycleValue);  // ref Robin2's Serial Output Basics
    Serial.print(',');   // http://forum.arduino.cc/index.php?topic=396450.0
    //    Serial.print(temperatureT1); // send temperature read from analog input temperaturePin
    Serial.print(get3231Temp()); // read DS3231 temperature
    Serial.print(',');
    Serial.print(batteryVoltage); // send battery voltage
    Serial.print(',');
    Serial.print(now.minute(), DEC); // send minute
    Serial.print(',');
    Serial.print(now.hour(), DEC); // send hour
    Serial.print(",A>"); // send sensor identifier
    Serial.println();


The pending test (ref cattledog) with use of A. is not yet shown here.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 01:56 pm
Would it not be much more elegant to simply add a ',' as a terminating character here
Code: [Select]
if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars-1) {
                    ndx = numChars - 2;             // and forcefully leave 1 extra character of space here
                }
            }
            else {
                receivedChars[ndx] = ',';          // put an extra separator
                receivedChars[ndx+1] = '\0';   // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }

The issue with that I think is that , are used as separator, and termination is done with endmarker: this allows the program to differentiate between "continuous" data and "last" data.
What advantage would the use of , as endmarker have?
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 15, 2019, 07:10 pm
Quote
What advantage would the use of , as endmarker have?
what was suggested earlier just so strtok() is looking for something that it can find.

Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged.  If that is the situation for this test, then there must be something wrong with the null termination.

Code: [Select]

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off


Actually, I have higher hopes for the addition of another token after the A.  It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.
my point also, when a '>' endmarker is received do not just terminate with a NULL but add a ',' separator as well. I can't see anything being wrong with the terminating NULL other then strtok() not finding it !
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 08:02 pm
what was suggested earlier just so strtok() is looking for something that it can find.
my point also, when a '>' endmarker is received do not just terminate with a NULL but add a ',' separator as well. I can't see anything being wrong with the terminating NULL other then strtok() not finding it !
Indeed, just to recap: now A is being sent without anything behind it, so placing a . or a , after the A and have strtok identify this character: strtokIndx = strtok(NULL, "."); in case of A.
Right?
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 15, 2019, 08:17 pm
Quote
Indeed, just to recap: now A is being sent without anything behind it, so placing a . or a , after the A and have strtok identify this character: strtokIndx = strtok(NULL, "."); in case of A.
Right?
I think the suggestion was to add the final token character in the receiving code instead of in the transmission. In Robin's code, the end marker is replace by '\0' and it was suggested that a separator be added before the Null. You could probably send the final separator in the transmission and add another at the receive.

I think the bigger issue is why strtok with the ESP8266 appears to not find the null terminator when parsing something at the end of the character string.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 15, 2019, 09:01 pm
I think the suggestion was to add the final token character in the receiving code instead of in the transmission. In Robin's code, the end marker is replace by '\0' and it was suggested that a separator be added before the Null. You could probably send the final separator in the transmission and add another at the receive.

I think the bigger issue is why strtok with the ESP8266 appears to not find the null terminator when parsing something at the end of the character string.

Thanks for that clarification cattledog. Tomorrow afternoon tests will be done, parameters and results I post asap.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 16, 2019, 05:34 pm
Actually this might be a clue if you have done this correctly. Did you only replace the last use of strtok for the message and the printing of the 6. The earlier cases should have been left unchanged.  If that is the situation for this test, then there must be something wrong with the null termination.

Code: [Select]

  strtokIndx = strtok(NULL, NULL);// this continues where the previous call left off
 //strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  strcpy(messageFromPC, strtokIndx); // this is a text string: copy it to messageFromPC:
  // sensor identification, sensor "A" or sensor "B", or C or D
  Serial.println(F(" 6 "));
  Serial.println();


Actually, I have higher hopes for the addition of another token after the A.  It could be a ',' or a '.' --just another character that strtok is looking for instead of the end of the string when looking for a non existent ','.
So far so good, I have been simulating several hours of transmissions/receptions, no crashes so far (in the previous setup there would have been several crashes). I keep this running another 24 hours (RX every 5 seconds) and let you know the result.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 03:23 pm
I think the suggestion was to add the final token character in the receiving code instead of in the transmission. In Robin's code, the end marker is replace by '\0' and it was suggested that a separator be added before the Null. You could probably send the final separator in the transmission and add another at the receive.

I think the bigger issue is why strtok with the ESP8266 appears to not find the null terminator when parsing something at the end of the character string.

After 24h testing (RX every 5 seconds instead of every 20 minutes: simulating 10 days in each hour of the past 24h), 2 crashes occured. An order of magnitude better than before the change according to cattledog suggestion.

But what can be done to eliminate these spurious crashes completely? The test is now run in ideal lab conditions, and I suspect the same issue being present because the crash still occurs when parsing the last part: the transmitter ID (A.. to D).
Is there a fallback for this last parsing instruction, ie building in a condition to somehow verify invalid ID characters?
Or would the consensus be that this result is "good enough"?

Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 17, 2019, 03:42 pm
Is there a fallback for this last parsing instruction, ie building in a condition to somehow verify invalid ID characters?
Or would the consensus be that this result is "good enough"?
Why not check the tail end of the data before starting to parse. Then you can add a suitable terminator if needed. Just make sure that the array into which you receive the data is ALWAYS a few characters larger than the largest message that may be received.


...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 05:06 pm
Why not check the tail end of the data before starting to parse. Then you can add a suitable terminator if needed. Just make sure that the array into which you receive the data is ALWAYS a few characters larger than the largest message that may be received.


...R
Good idea. Can you give some clue how this can be done without parsing please?
Currently the data to be parsed sits in tempchars[numChars] where numChars is 32. The data sent is in this format: <47.50,12.50,6.50,5,10,A> (hence: 3 floats, 2 integers and one character A or B or C or D).

EDIT: the format currently tested is with A. hence: <47.50,12.50,6.50,5,10,A.>
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 17, 2019, 05:43 pm

Quote
After 24h testing (RX every 5 seconds instead of every 20 minutes: simulating 10 days in each hour of the past 24h), 2 crashes occured
In your testing, did you add the extra terminator to the transmission and again at the receive when the end marker is found?

Quote
<47.50,12.50,6.50,5,10,A>
Are the transmitted values always that length, or can the number of digits in the floats or integers change?

If the length is constant, and I count correctly, tempchars[23] should be the letter, tempchars[24] should be the final separator(',' or '.'), and tempchars[25] should be '\0'.

Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 06:21 pm
In your testing, did you add the extra terminator to the transmission and again at the receive when the end marker is found?
Yes.

This is the code in the transmitter used to send the packet of data:
Code: [Select]

    Serial.print('<'); // this section for HC-12 transmission
    Serial.print(cycleValue);  // ref Robin2's Serial Output Basics
    Serial.print(',');   // http://forum.arduino.cc/index.php?topic=396450.0
    //    Serial.print(temperatureT1); // send temperature read from analog input temperaturePin
    Serial.print(get3231Temp()); // read DS3231 temperature
    Serial.print(',');
    Serial.print(batteryVoltage); // send battery voltage
    Serial.print(',');
    Serial.print(now.minute(), DEC); // send minute
    Serial.print(',');
    Serial.print(now.second(), DEC); // send hour
    Serial.print(",A.>"); // send sensor identifier
    Serial.println();


This is the code at the receiver to parse the last section (the identifier A, or B, or C or D):

Code: [Select]

  strtokIndx = strtok(NULL, "."); // identifier parsing



Are the transmitted values always that length, or can the number of digits in the floats or integers change?

If the length is constant, and I count correctly, tempchars[23] should be the letter, tempchars[24] should be the final separator(',' or '.'), and tempchars[25] should be '\0'.
Yes, the length always is constant.

Ok, I try that suggestion; the reason I kept 32 is because at the time this was still in prototype testing. By now I know this length will not alter anymore.




Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 17, 2019, 06:43 pm
It's interesting that there are no other errors with the data except for the tail end. More robust error checking with check sums, could be incorporated, and might not be a bad idea.

As a quick test with your current code, to both fix the error and see where it is, you could try the following

Code: [Select]

if (newData == true) {
  if (receivedChars[24] != '.')
  {
    Serial.println("Missing .");
    receivedChars[24] = '.';
  }
  if (receivedChars[25] != '\0')
  {
    receivedChars[25] = '\0';
    Serial.println("Missing NULL");
  }
  strcpy(tempChars, receivedChars);
  // this temporary copy is necessary to protect the original data
  //   because strtok() used in parseData() replaces the commas with \0
  //
  // Serial.println(receivedChars); // serial monitor output; not usefull on ESP8266
  // Serial.println();
  parseData();
  // useParsedData(); // still issue with execution and timing (9/5/2018); solved: use in irrigationCommand();
  // thingSpeak();
  newData = false;
}
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 07:19 pm
It's interesting that there are no other errors with the data except for the tail end. More robust error checking with check sums, could be incorporated, and might not be a bad idea.

As a quick test with your current code, to both fix the error and see where it is, you could try the following

Code: [Select]

if (newData == true) {
  if (receivedChars[24] != '.')
  {
    Serial.println("Missing .");
    receivedChars[24] = '.';
  }
  if (receivedChars[25] != '\0')
  {
    receivedChars[25] = '\0';
    Serial.println("Missing NULL");
  }
  strcpy(tempChars, receivedChars);
  // this temporary copy is necessary to protect the original data
  //   because strtok() used in parseData() replaces the commas with \0
  //
  // Serial.println(receivedChars); // serial monitor output; not usefull on ESP8266
  // Serial.println();
  parseData();
  // useParsedData(); // still issue with execution and timing (9/5/2018); solved: use in irrigationCommand();
  // thingSpeak();
  newData = false;
}

Checksums: indeed may become necessary.
Your suggestion above: will be done now, checking back in a few days.
Thank you cattledog.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 17, 2019, 09:09 pm
EDIT: the format currently tested is with A. hence: <47.50,12.50,6.50,5,10,A.>
Then it seems that you should check that the final character is a full-stop.

IF the final character is a full-stop AND the program crashes sometimes then I don't know what is causing it. The ESP8266 is a lot more complex than an Uno because it has all that WiFi code that needs to run at regular intervals. Is there any chance that something in your code (maybe nothing to do with parsing) is interfering with the WiFi stuff and the crash is actually happening somewhere else?

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 09:51 pm
Then it seems that you should check that the final character is a full-stop.

IF the final character is a full-stop AND the program crashes sometimes then I don't know what is causing it. The ESP8266 is a lot more complex than an Uno because it has all that WiFi code that needs to run at regular intervals. Is there any chance that something in your code (maybe nothing to do with parsing) is interfering with the WiFi stuff and the crash is actually happening somewhere else?

...R
I also thought about that until this stripped down version of my program (ref. to your helpfull suggestion) was tested before cattledog's suggestion of A. ; but then too I got these crashes. And I use the yield; command to allow these background processes to perform. After implementing cattledog's A. suggestion there was a major improvement.

I have an ESP32 where this program compiles correctly, but the hardware is not set up yet for the HC12 radio connection; after that is ready, and after testing all of the current suggestions, that is going to be plan B.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 17, 2019, 10:28 pm
Just a few minutes into testing the results are worse than before:
"Missing ." shows up every second or third or fourth parse from identifier B, however the sent message does contain B. so in the sent message the output is correct.
But every time after "Missing ." shows up the ESP8266 crashes.
Now: revert back to the previous setup, without the "Missing ." check and see what happens the next 24h. To make sure nothing else is at play.

Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 17, 2019, 11:45 pm
Just a few minutes into testing the results are worse than before:
"Missing ." shows up every second or third or fourth parse from identifier B, however the sent message does contain B. so in the sent message the output is correct.
The results may be worse but the cause of the problem may be in sight.

Post the program.

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 07:33 am
The results may be worse but the cause of the problem may be in sight.

Post the program.

...R
Here is the program including the latest terminator verification/rectification as suggested by cattledog.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 18, 2019, 09:36 am
This code
Code: [Select]
  if (newData == true) {
   
    if (receivedChars[24] != '.') // verify if the last section contains a valid terminator . before the endmarker
    {
      Serial.println("Missing .");
      receivedChars[24] = '.';
    }
   
    strcpy(tempChars, receivedChars);

is not checking the array that is being parsed.

Also you can't be certain that the end of the array is at index 24. You need to check the length to get an index to the end - remembering that the last element will be [length-1]

...R
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 18, 2019, 10:36 am
Honestly i think by now i would want to get rid of the strtok() function within the parsing all-together, and create my own version of it, that copies and adds to a char* (or even a String, hey you are using an ESP.. ) until it finds a separator or a terminator. This is patch work.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 11:13 am
This code
Code: [Select]
 if (newData == true) {
    
    if (receivedChars[24] != '.') // verify if the last section contains a valid terminator . before the endmarker
    {
      Serial.println("Missing .");
      receivedChars[24] = '.';
    }
    
    strcpy(tempChars, receivedChars);

is not checking the array that is being parsed.

Also you can't be certain that the end of the array is at index 24. You need to check the length to get an index to the end - remembering that the last element will be [length-1]

...R
Right, the check should be on tempchars, not receivedchars:
Code: [Select]

if (tempchars[24] != '.') // verify if the last section contains a valid terminator . before the endmarker
    {
      Serial.println("Missing .");
      tempchars[24] = '.';
    }


Indeed, looking at the serial output it seems that the . is not where it was supposed to be, especially because placing a . at that position causes a crash.
So before performing above verification there also needs to be a length verification (although I would not know why the length is not constant given the specific nature of the data being sent).
Question: how to do this length check? My knowledge of string constructs is next to 0, sorry!

Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 11:16 am
Honestly i think by now i would want to get rid of the strtok() function within the parsing all-together, and create my own version of it, that copies and adds to a char* (or even a String, hey you are using an ESP.. ) until it finds a separator or a terminator. This is patch work.
I appreciate your input, but my knowledge of writing parsing code is next to 0. And since the work by Robin2 is a major contribution to this forum I would not want to bypass him for this subject.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 18, 2019, 01:42 pm
So before performing above verification there also needs to be a length verification (although I would not know why the length is not constant given the specific nature of the data being sent).
Question: how to do this length check? My knowledge of string constructs is next to 0, sorry!
You have to keep in mind that the Arduino has no control over the data it receives. There could be interference with the signal that causes bytes to be misinterpreted (not likely, but possible). And at any one time the received data might include bytes that arrived after the terminator for the previous message. For example if the terminator was '>' and a carriage-return and line-feed followed it.

You can use the strlen() (http://www.cplusplus.com/reference/cstring/strlen/) function to determine the length - but read the documentation carefully so you know exactly what it is telling you. Then, if you feel it is necessary, you could iterate over the entire message byte by byte to make sure there are no unexpected values. Have you tried printing the received message just before parsing it?

I can think of no good reason to abandon the strtok() function. It has been tried and tested and found satisfactory millions of times.

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 02:17 pm
You have to keep in mind that the Arduino has no control over the data it receives. There could be interference with the signal that causes bytes to be misinterpreted (not likely, but possible). And at any one time the received data might include bytes that arrived after the terminator for the previous message. For example if the terminator was '>' and a carriage-return and line-feed followed it.

You can use the strlen() (http://www.cplusplus.com/reference/cstring/strlen/) function to determine the length - but read the documentation carefully so you know exactly what it is telling you. Then, if you feel it is necessary, you could iterate over the entire message byte by byte to make sure there are no unexpected values. Have you tried printing the received message just before parsing it?
Good idea. Will be done this evening, results in a day.

This code I will insert right before the parsing starts:

Code: [Select]

  for(int i = 0; i < 32; i++)
  {
    Serial.print(tempChars[i]);
  }
  Serial.println();




I can think of no good reason to abandon the strtok() function. It has been tried and tested and found satisfactory millions of times.

...R
Your contribution on the subject I consider the best I found anywhere.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 18, 2019, 02:59 pm
This code I will insert right before the parsing starts:

Code: [Select]

  for(int i = 0; i < 32; i++)
  {
    Serial.print(tempChars[i]);
  }
  Serial.println();



If you add the line
Serial.print(' ');
to put a space between characters it may make the data easier to read - unless there are space characters in the data

Another idea is to change
Code: [Select]
Serial.print( tempChars[1] );
to
Code: [Select]
Serial.print( (byte) tempChars[i] );
which will show the Ascii values in case there are some non-printing characters

...R
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 18, 2019, 03:18 pm
I can think of no good reason to abandon the strtok() function. It has been tried and tested and found satisfactory millions of times.
But now it is causing a problem since it reads beyond the terminating NULL. (as far as we can tell) So i would want to add a 'do not read beyond size of string' to it, as far as we know it is the cause of the crashes (on the ESP)
Quote
since the work by Robin2 is a major contribution to this forum
I totally agree with that. I also find the Serial transmit-receive examples very good ! (though incomplete imo.)
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 03:47 pm
If you add the line
Serial.print(' ');
to put a space between characters it may make the data easier to read - unless there are space characters in the data

Another idea is to change
Code: [Select]
Serial.print( tempChars[1] );
to
Code: [Select]
Serial.print( (byte) tempChars[i] );
which will show the Ascii values in case there are some non-printing characters

...R
Great!! I will serial print both.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 03:49 pm
But now it is causing a problem since it reads beyond the terminating NULL. (as far as we can tell) So i would want to add a 'do not read beyond size of string' to it, as far as we know it is the cause of the crashes (on the ESP)
Ok, that is what we are trying to figure out now for sure, see my previous post; results in a day or so.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 18, 2019, 04:19 pm
But now it is causing a problem since it reads beyond the terminating NULL.
You can bet the farm that it isn't - unless doing is part of its defined behaviour.

...R
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 18, 2019, 04:46 pm
You can bet the farm that it isn't - unless doing is part of its defined behaviour.
I know, but then what is causing the crashes ? anyway time will tell.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 18, 2019, 06:48 pm
I know, but then what is causing the crashes ? anyway time will tell.
I don't know either, but pfaffing about with strtok() would just be a waste of time. It's what I call the "scatter gun" approach to debugging - "do something - ANYTHING"

...R
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 18, 2019, 07:38 pm
I don't know either, but pfaffing about with strtok() would just be a waste of time. It's what I call the "scatter gun" approach to debugging - "do something - ANYTHING"
I thought there were totally acceptable solutions presented earlier (Adding a separator after reception of the terminator '>') and on top of that nobody has given any explanation for the strtok(NULL,NULL) not working properly. Now the solution seems to be "Trail for a long time and wait for Error"
We are going from the point of view that this is just a partial test sketch but  i haven't seen a complete sketch anywhere (i may have missed it) and nobody complained about it.  I do hope there is not a part where interrupts are turned off, since Serial reception does depend on it being turned on. I would have started parsing manually without strtok() even if it was just to exclude that the list of possible issues. (and also because i don't think it would be to hard to do.)
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 18, 2019, 07:49 pm
Quote
on top of that nobody has given any explanation for the strtok(NULL,NULL) not working properly.
My thinking has been that either the strtok implementation in the esp8266 core has a problem, or, more likely in my opinion, that the NULL is somehow missing. Perhaps because the total received chars overruns the 32 character boundary.

I think that the idea to print the ascii value of everything received as it arrives is a good debugging idea.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 07:49 pm
...I do hope there is not a part where interrupts are turned off, since Serial reception does depend on it being turned on...
To prevent any fog clouding the issue at hand: my post #34 contains the bare-bones program with all essentials related to reception and parsing available. No interrupts are disabled anywhere, not in the original, not in the current bare-bones program.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 08:24 pm
My thinking has been that either the strtok implementation in the esp8266 core has a problem, or, more likely in my opinion, that the NULL is somehow missing. Perhaps because the total received chars overruns the 32 character boundary.

I think that the idea to print the ascii value of everything received as it arrives is a good debugging idea.
I just got home, downloaded the modified receiving/parsing program (see annex). This program expects a dot after the TX identifier, which is being sent too by the transmitter. Here are the results of the first readings:

When no missing dot reported by RX

parse data() actual: 4 6 . 9 5 , 2 0 . 7 5 , 9 . 4 9 , 1 , 7 , B .   .
parseData() ASCII : 52 54 46 57 53 44 50 48 46 55 53 44 57 46 52 57 44 49 44 55 44 66 46 0 46 0 0 0 0 0 0 0


Actually parsed data: 46.95 : 20.75 : 9.49 : 1 : 7 : B

When missing dot reported by RX

parseData() actual : 4 6 . 9 6 , 2 0 . 7 5 , 9 . 5 7 , 0 , 4 7 , B . .
parseData() ASCII : 52 54 46 57 54 44 50 48 46 55 53 44 57 46 53 55 44 48 44 52 55 44 66 46 46 0 0 0 0 0 0 0

Actually parsed data: 46.96 : 20.75 : 9.57 : 0 : 47 : B



What surprises me at first view is that
1. more often than not there are missing dot and even though it is not missing, a dot is being added by the RX program, as written on lines 52 to 56.
2. when no missing dot is reported, still 2 dots appear in the actual message albeit with a space between inside the message (noted as a 0 in the ASCII part)

Now this is the program using "missing dot detection" so this is the one that crashes a lot.
When removing he "missing dot program section" (lines 52 to 56) these crashes become very rare.

But then the output becomes (no "missing-dot-detection"):

parseData() actual : 4 3 . 3 1 , 2 1 . 0 0 , 9 . 5 7 , 2 0 , 5 7 , D .

parseData() ASCII : 52 51 46 51 49 44 50 49 46 48 48 44 57 46 53 55 44 50 48 44 53 55 44 68 46 0 0 0 0 0 0 0

RX parsed message = D
43.31 : 21.00 : 9.57 : 20 : 57 : D

No double dots at the end of the actual message.

Note: the first float is a duty cycle reading, the second float is a temperature reading, the third float is a battery voltage reading, the next is an integer for the minute when the TX took place, the next integer is the second when the TX took place and the last is the TX identifier.

I would let this run for 24 hours and report the monitor results if/when crashes occured.
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 18, 2019, 08:53 pm
I'm afraid I messed up here
Quote
Quote
<47.50,12.50,6.50,5,10,A>
Are the transmitted values always that length, or can the number of digits in the floats or integers change?

If the length is constant, and I count correctly, tempchars[23] should be the letter, tempchars[24] should be the final separator(',' or '.'), and tempchars[25] should be '\0'.
I forgot that Robin2's routine strip both the start and end markers from the message.

The '.' should be at tempchars[23]
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 08:54 pm
I'm afraid I messed up here
I forgot that Robin2's routine strip both the start and end markers from the message.

The '.' should be at tempchars[23]

Thank you cattledog, I will test again tomorrow, after this 24h run.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 09:01 pm
Something strange appeared in the serial monitor just now:

Message x:

actual message content: 4 0 . 8 2 , 2 4 . 2 5 , 7 . 6 5 , 4 1 , 2 , A .
ASCII message content: 52 48 46 56 50 44 50 52 46 50 53 44 55 46 54 53 44 52 49 44 50 44 65 46 0 0 0 0 0 0 0 0
Parsed message: 40.82 : 24.25 : 7.65 : 41 : 2 : A


Message x+1:
actual message content: 4 6 . 9 6 , 2 0 . < 4 3 . 3 2 , 2 0 . 7 5 , 9 . 5 7 , 4 1 , 1
ASCII message content: 52 54 46 57 54 44 50 48 46 60 52 51 46 51 50 44 50 48 46 55 53 44 57 46 53 55 44 52 49 44 49 0
Parsed Message content: 46.96 : 20.00 : 20.75 : 9 : 41 : 1



Message x+2:
actual message content:  4 0 . 8 1 , 2 4 . 2 5 , 8 . 0 9 , 4 1 , 2 2 , A .       41  1
ASCII message content: 52 48 46 56 49 44 50 52 46 50 53 44 56 46 48 57 44 52 49 44 50 50 44 65 46 0 0 52 49 0 49 0
Parsed message: 40.81 : 24.25 : 8.09 : 41 : 22 : A


In subsequent messages the identifier keeps being correct: both in the full actual message as in the parsed message. BUT in the actual message content the trailing 41 and 1 keep being present.

In message x+1 in the second float there suddenly appears a < within this number.
In the serial monitor the positioning of this 41 and 1 are always exacty identical, meaning they do appear in the same exact position too in the received message?
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 18, 2019, 09:04 pm
Quote
the next is an integer for the minute when the TX took place, the next integer is the second when the TX took place
I think that these time values can be either one or two digits, and the character array length can vary. You may want to sent the values with a leading 0 so they are always 2 digits, or else determine the actual length of what is received.

I think you will now be back go the situation where you have the rare crashes. I'm sure we can figure out what is going on, now that the walk in the swamp due to my incorrect debug suggestion is over.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 09:06 pm
I think that these time values can be either one or two digits, and the character array length can vary. You may want to sent the values with a leading 0 so they are always 2 digits, or else determine the actual length of what is received.
You might very well have hit the nail on its ugly head!

I think you will now be back go the situation where you have the rare crashes. I'm sure we can figure out what is going on, now that the walk in the swamp due to my incorrect debug suggestion is over.
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 18, 2019, 09:16 pm
Quote
In message x+1 in the second float there suddenly appears a < within this number.
Are you certain that with your accelerated testing you are not sending a message before the previous one is processed? You may be getting two message strings mixed together.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 09:19 pm
Are you certain that with your accelerated testing you are sending a message before the previous one is processed? You may be getting two message strings mixed together.
Yes, I can clearly see on the monitor the progress of the RX and parsing, and the timing between the 3 transmitters is exactly 5 seconds (the RTC's are calibrated down to the second).
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 18, 2019, 09:24 pm
I think that these time values can be either one or two digits, and the character array length can vary. You may want to sent the values with a leading 0 so they are always 2 digits, or else determine the actual length of what is received.

I think you will now be back go the situation where you have the rare crashes. I'm sure we can figure out what is going on, now that the walk in the swamp due to my incorrect debug suggestion is over.
The strange thing is that message x+1 indeed shows up on a transition from second 57 to second 2, but in previous messages this transition did occur already and then no issue showed up with this.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 18, 2019, 11:58 pm
One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
Second if the startmarker ends up in the middle of the message (x+1), something is really wrong ! maybe the buffer was not completely empty when the new message was sent.
I don't see  a problem with the residual bytes in x+2, they are behind a null terminator.
Quote
My thinking has been that either the strtok implementation in the esp8266 core has a problem, or, more likely in my opinion, that the NULL is somehow missing. Perhaps because the total received chars overruns the 32 character boundary.
both are possible, but since we are not low on memory at the moment how about increasing the buffer to something that is easily enough(100 bytes ?) until we can figure out what is going on.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 19, 2019, 12:24 am
To prevent any fog clouding the issue at hand: my post #34 contains the bare-bones program with all essentials related to reception and parsing available. No interrupts are disabled anywhere, not in the original, not in the current bare-bones program.
but not within </> code tags. that is why i missed it.
what is 'wire' being used for ?
also i think the corruption of the data may have something to do with
Code: [Select]
if (ndx >= numChars) {
          ndx = numChars - 1;
        }
which prevents writing beyond the size of the array, but every new character will just overwrite the previous one. The test is valid, but the error handling is insufficient.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 05:55 am
One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
But it currently results in a order of magnitude reduction in crashes.

Second if the startmarker ends up in the middle of the message (x+1), something is really wrong ! maybe the buffer was not completely empty when the new message was sent.
How to find out?

I don't see  a problem with the residual bytes in x+2, they are behind a null terminator. both are possible, but since we are not low on memory at the moment how about increasing the buffer to something that is easily enough(100 bytes ?) until we can figure out what is going on.
But why did they get there?

These residual bytes 41 and 1 are still there after about 4320 runs at this moment.

Should I restart with a buffer increased from 32 to your suggested 100?
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 05:58 am
but not within </> code tags. that is why i missed it.
what is 'wire' being used for ?
The wire.h is used in the original program to accomodate use of I2C

also i think the corruption of the data may have something to do with
Code: [Select]
if (ndx >= numChars) {
          ndx = numChars - 1;
        }

which prevents writing beyond the size of the array, but every new character will just overwrite the previous one. The test is valid, but the error handling is insufficient.
How would you suggest to improve this?
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 19, 2019, 07:58 am
Quote
One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
The '.' is not used as an end marker. The start marker is '<' and the end marker is '>' . The '.' is a separator. The way you manage strtok, it is not going to be a separator for its earlier appearances in the data of the array.

Quote
These residual bytes 41 and 1 are still there after about 4320 runs at this moment.
I'm not sure how they got there, but there is nothing to clear the buffer, and since the buffer of 32 is larger than the received data those characters will just sit there. You can clear the buffer with this

Code: [Select]
if (newData == true) {
    memset(tempChars, '\0', sizeof(tempChars));
    strcpy(tempChars, receivedChars);
    //rest of code


Quote
Should I restart with a buffer increased from 32 to your suggested 100?
I think this a good idea to help see if there are rare cases where the received characters over run the 32 character buffer and the final separator gets overwritten.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 08:40 am
...

I'm not sure how they got there, but there is nothing to clear the buffer, and since the buffer of 32 is larger than the received data those characters will just sit there. You can clear the buffer with this

Code: [Select]
if (newData == true) {
    memset(tempChars, '\0', sizeof(tempChars));
    strcpy(tempChars, receivedChars);
    //rest of code


I think this a good idea to help see if there are rare cases where the received characters over run the 32 character buffer and the final separator gets overwritten.
Done. Buffer increased from 32 to 64, to keep the monitor screen display within manageable size. Downloaded and restarted. I insert a yield(); command in the Serial.print output to prevent, at 1200 baud, the ESP8266 crashing on watchdog timer.

I will let this run for a day or so. The individual transmissions still take place with exactly 5 second spacings between D, A and B, and 10 seconds between B and D (C currently not used).

Question 1: the appearance of < in the message x+1 still puzzles me. Is there any possible explanation?

Question 2: and the code below, is its purpose to reset the ndx counter in case no endmarker is received by the end of the 32 byte buffer? And if that is the case then the else after that if is not executed?
I refer to message #61 from Deva_Rishi.

Code: [Select]

      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 19, 2019, 09:02 am
Quote
Question 1: the appearance of < in the message x+1 still puzzles me. Is there any possible explanation?
Not that I have been able to come up with other than overlapping messages which you say can't occur.

Quote
Question 2: and the code below, is its purpose to reset the ndx counter in case no endmarker is received by the end of the 32 byte buffer?
It does not reset the ndx counter, but holds it at its max value so that no characters are written to a memory location outside of the array range.

Quote
And if that is the case then the else after that if is not executed?
No, the else condition is executed when the end marker is received.

Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 19, 2019, 09:33 am
These residual bytes 41 and 1 are still there after about 4320 runs at this moment.
When a new message arrives which is shorter than the previous message the tail of the previous data can remain in the buffer. Normally that is never noticeable because the cstring functions never look past the terminating NULL from the current message - in other words the first NULL that is detected.

You could adapt the code to specifically write space characters into every place in the buffer before starting to receive a new message, but I have never found it necessary.

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 01:25 pm
During the 6 hours the test currently runs one crash occured:

actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .

ASCII message: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Parsing crashed at the fifth character which is not present in the actual message

During this run there never was anything written beyond the 23 or 24 bytes being expected.

Clearly this one crash is due to a corrupted byte received.

Maybe some kind of message integrity check needs to be included in the sent message? If so, how best to program?


PS: a joke I just heard from my wife: "The oldest computer was an Apple, owned by Adam and Eve. It had very limited memory, just one bite and it crashed".
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 19, 2019, 02:38 pm
During the 6 hours the test currently runs one crash occured:

actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .

ASCII message: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Parsing crashed at the fifth character which is not present in the actual message
What do you mean by the 5th character? I reckon it is the 255 (which is the 8th character) that is causing the problem

A very simple integrity check is to XOR all the bytes and include the XOR value as the last byte. However that would mean that your system would have to be able to handle any value from 0 to 255 so that the use of < and > as start and end markers would not work. Maybe the simplest solution is to Serial.print() the XOR value with suitable padding to ensure it is always 3 bytes long.

Remind us what is sending the data and what baud rate is being used?

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 02:53 pm
What do you mean by the 5th character? I reckon it is the 255 (which is the 8th character) that is causing the problem

A very simple integrity check is to XOR all the bytes and include the XOR value as the last byte. However that would mean that your system would have to be able to handle any value from 0 to 255 so that the use of < and > as start and end markers would not work. Maybe the simplest solution is to Serial.print() the XOR value with suitable padding to ensure it is always 3 bytes long.

Remind us what is sending the data and what baud rate is being used?

...R
The baud rate used is 1200: this allows the maximum possible range for the HC12 radio.

The transmitted message is composed as follows:

Code: [Select]


    Serial.print('<'); // this section for HC-12 transmission
    Serial.print(cycleValue);  // ref Robin2's Serial Output Basics
    Serial.print(',');   // http://forum.arduino.cc/index.php?topic=396450.0
    //    Serial.print(temperatureT1); // send temperature read from analog input temperaturePin
    Serial.print(get3231Temp()); // read DS3231 temperature
    Serial.print(',');
    Serial.print(batteryVoltage); // send battery voltage
    Serial.print(',');
    Serial.print(now.minute(), DEC); // send minute
    Serial.print(',');
    Serial.print(now.second(), DEC); // send hour
    Serial.print(",A.>"); // send sensor identifier
    Serial.println();


where cycleValue, temperatureT1, batteryVoltage are floats, and now.minute() and now.second() are integers derived from the RTClibExtended.h library.

Sorry my mistake for stating the 5th character: you are correct.

Serial.print the XOR value : how should I include this in the transmitted serial.print?
Thanks!!

Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 19, 2019, 06:15 pm
The baud rate used is 1200: this allows the maximum possible range for the HC12 radio.

Serial.print the XOR value : how should I include this in the transmitted serial.print?

Just put it in before the closing >

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 06:20 pm
Just put it in before the closing >

...R
Can you please help writing this XOR calculation and writing?
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 19, 2019, 06:33 pm
Quote
actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .
Quote
Parsing crashed at the fifth eight character which is not present in the actual message

During this run there never was anything written beyond the 23 or 24 bytes being expected.

Clearly this one crash is due to a corrupted byte received.
Before getting into error checking and checksums, I want to understand a bit more about what is going on.

Please describe more about the "crash". Did the ESP freeze, reset, ?

There is a lot more wrong with this data than an incorrect character. There are a missing 9 or 10 bytes. A simple test for message length would have picked this up. It looks like thee was a failure of the temperature and voltage readings. Perhaps they can be tested for correctness when measured.

Is this the first time that the print out has not reached "5" and the crash occurs at the last parsing?

I will test, but I think that strtok should have handled the mangled string without crashing on a UNO and just reported strange or NULL values.

The behaviour of strtok in the esp8266 environment continues to interest me.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 06:54 pm
Before getting into error checking and checksums, I want to understand a bit more about what is going on.

Please describe more about the "crash". Did the ESP freeze, reset, ?
The parsing index went to 4, meaning that when parsing the 4th value (the first integer after 3 floats) it went good, and the crash occured at next value, the second integer:

actual data: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .
ASCII data: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 1  2  3  4

Then "exception 28" occured, a stack dump showed up, and the ESP8266 automatically restarted and continued.


There is a lot more wrong with this data than an incorrect character. There are a missing 9 or 10 bytes. A simple test for message length would have picked this up. It looks like thee was a failure of the temperature and voltage readings. Perhaps they can be tested for correctness when measured.
Interesting. What I will do is limit to one transmitter and connect that too to a serial monitor and readout whatever is being transmitted.

Is this the first time that the print out has not reached "5" and the crash occurs at the last parsing?
Yes, at least on this run (10hrs so far with this one crash).

I will test, but I think that strtok should have handled the mangled string without crashing on a UNO and just reported strange or NULL values.

The behaviour of strtok in the esp8266 environment continues to interest me.
That pleases me; the ESP8266 has so much potential it would be nice to get this sorted out.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 19, 2019, 07:58 pm
The wire.h is used in the original program to accomodate use of I2C
Well while we are not using it, comment it out

How would you suggest to improve this?
at the very least let's send information about how often this situation occurs to the Serial for now.

Quote
Question 1: the appearance of < in the message x+1 still puzzles me. Is there any possible explanation?
The explanation that somehow there may be overlapping messages, due to an endmarker not being received.
Quote
During the 6 hours the test currently runs one crash occured:

actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .

ASCII message: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
255 ?? that was never sent, your transmission-reception is corrupted ! I want to know your complete hardware setup. How long are your cables how are they powered and how are they shielded ? I was going from the point of view that you had it in a 'lab' setup using the low BAUD-rate but not depending on it, but now i am no longer sure. If the bits being received are not the same as the ones being sent then all the other work and suggestions are pointless other than for educational purpose.
Title: Re: Serial Input Basics: example 5, question
Post by: Robin2 on Jan 19, 2019, 10:11 pm
As far as I can see whatever is the problem it is not a fault with my Serial Input Basics (http://forum.arduino.cc/index.php?topic=396450.0)

There are too many cooks paddling around in this soup so I'm gonna drop out.

Just one last thought ... any time I have found myself with a significant programming problem the solution has ALWAYS involved making things simpler and shorter.

...R
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 19, 2019, 10:34 pm
As far as I can see whatever is the problem it is not a fault with my Serial Input Basics (http://forum.arduino.cc/index.php?topic=396450.0)

There are too many cooks paddling around in this soup so I'm gonna drop out.

Just one last thought ... any time I have found myself with a significant programming problem the solution has ALWAYS involved making things simpler and shorter.

...R
I am sorry to read this but I understand. Sad because this is not of my making. Can you please give your input how to program the calculated XOR at transmission and at reception?
Title: Re: Serial Input Basics: example 5, question
Post by: Whandall on Jan 20, 2019, 12:48 am
Can you please give your input how to program the calculated XOR at transmission and at reception?
Is Google down?

If it comes up again you could try https://www.google.com/search?q=xor+checksum+example+c%2B%2B (https://www.google.com/search?q=xor+checksum+example+c%2B%2B)

Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 21, 2019, 10:36 pm
I talked about manually parsing the data before, and well sometimes it is time to put your time where your mouth is, and so here with the inclusion of Robin's data reception i came up with this bit of code which will just invalidate any input that was not within it's parameters (and not using strtok() )
Quote
Code in attachment due to 9000 chars limit
I also included the hexvalue, that you wanted in your most recent post, so by now there are 7 data pieces, it is more for educational purpose, since as far as i know you will get to working version without it, which brings me to this
Quote
The baud rate used is 1200: this allows the maximum possible range for the HC12 radio.
If you want to run a long line and because of that you drop down the BAUD-rate that far you should really look at improving the hardware. I suggest one of two options, My choice would be to make use of max485 standard, where by means of 2 transcievers (per direction of transmission) and a twisted wire you can cover distance of up to 300m at 250kbps
(https://forum.arduino.cc/index.php?action=dlattach;topic=590798.0;attach=291186)
Or to transmit using the same hardware specs as DALI uses and transmit and receive using Manchester, which is also just as slow at 1200 bps, but since it doesn't depend on a clock and is a lot less sensitive to sync issues then normal serial. Like that you would also be able to cover up to 150m. The second option is more complex to program and therefore i suggest the first  and since you only have data going 1 way a few parts and a UTP cable will do the trick and will provide reliable transmission.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 22, 2019, 10:06 am
Hi Deva_Rishe, thank you for your input.
However, in my setup communication takes place wirelessly with HC-12 433MHz radios: up to four transmitters, one receiver.
So far my tests have shown that the issue with crashing at the receiver most likely occurs because an invalid character did enter the sent data (or the very unlikely -but not proven untrue- case of 2 messages sent simultaneously), and not due to actual decoding or parsing at the receiver end.
Latest test result shows the equivalent of one crash during about 35.000 transmissions (in lab conditions).
As a result and after advice from Robin2 I added a XOR checksum calculation stored as 2 hex values in one int variable, and this is added after the start marker upon transmission.
What needs to be done now is to extract/remove these 2 hex values and the first comma in the received message. Then calculate the XOR checksum again, on the received character array. And continue accordingly.

But this extends beyond the subject of this threat.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 22, 2019, 10:33 am
communication takes place wirelessly with HC-12 433MHz radios: up to four transmitters, one receiver.
Ah i see you are using those, doesn't the virtual-wire library do a full CRC check ? as far as i know using that only allows for valid transmissions.

As a result and after advice from Robin2 I added a XOR checksum calculation stored as 2 hex values in one int variable, and this is added after the start marker upon transmission.
a 16-bit to verify seems cool.

What needs to be done now is to extract/remove these 2 hex values and the first comma in the received message. Then calculate the XOR checksum again, on the received character array.
With what i saw from the other thread, and/or with the info within my sketch, you should be able to manage that no problem.
Still you should be using the Virtual-wire library ! man those  433Mhz radios are far from reliable to begin with, and if they jitter long enough on your input line they may even cause your wdt to fire.
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 22, 2019, 12:51 pm
...
Still you should be using the Virtual-wire library ! man those  433Mhz radios are far from reliable to begin with, and if they jitter long enough on your input line they may even cause your wdt to fire.
35000 transmissions correct and one wrong is not too bad for an HC-12. Of course this is in lab conditions; that is why the checksum will be included.
CRC check on HC12: the datasheet does not mention it.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 22, 2019, 01:15 pm
it's not bad, but still Virtual Wire (https://www.pjrc.com/teensy/td_libs_VirtualWire.html)will take care of the CRC check for you, pity you didn't make it overly clear what your setup was (or i missed it within your initial post) we could have skipped a lot of issues. VirtualWire can even make 'Raw' data reliable.
Title: Re: Serial Input Basics: example 5, question
Post by: cattledog on Jan 22, 2019, 06:35 pm
Quote
Virtual Wirewill take care of the CRC check for you
True. RadioHead and VirtualWire have a  CCITT CRC16 baked in, but is would be simple to add this to the serial protocol of the HC12. There are arduino libraries for this, and I think they would be valid for the ESP8266.

The HC12's are designed to run with a serial UART, and I would be interested to know if @Deva_Rishi has direct experience running them with an ASK protocol and VirtualWire/RadioHead

Quote
As a result and after advice from Robin2 I added a XOR checksum calculation stored as 2 hex values in one int variable, and this is added after the start marker upon transmission.
What checksum routine are you using? I thought it was a simple byte wise XOR which should give an 8 bit checksum. It its not a CRC.

Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 22, 2019, 07:15 pm
The HC12's are designed to run with a serial UART, and I would be interested to know if @Deva_Rishi has direct experience running them with an ASK protocol and VirtualWire/RadioHead
I had them running using the VW-library using more or less any pin ( i had 12 & 3) at 4000bps of course it is a blocking protocol, (for sure for tx one should wait until transmission is complete, think it does it using a timer interrupt) but the sheer simplicity of it is remarkable (sketches upon request) I don't know for what they were designed, but i tried them out and for me the 1-way communication, and the total lack of security meant that i decided that i would not use them for anything other then controlling kids-toys
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 22, 2019, 09:05 pm
True. RadioHead and VirtualWire have a  CCITT CRC16 baked in, but is would be simple to add this to the serial protocol of the HC12. There are arduino libraries for this, and I think they would be valid for the ESP8266.

The HC12's are designed to run with a serial UART, and I would be interested to know if @Deva_Rishi has direct experience running them with an ASK protocol and VirtualWire/RadioHead

What checksum routine are you using? I thought it was a simple byte wise XOR which should give an 8 bit checksum. It its not a CRC.


The test code is attached: indeed not CRC, simple XOR calculation. 8 bit? I thought the output is 2 hex values?
Title: Re: Serial Input Basics: example 5, question
Post by: brice3010 on Jan 22, 2019, 09:09 pm
I had them running using the VW-library using more or less any pin ( i had 12 & 3) at 4000bps of course it is a blocking protocol, (for sure for tx one should wait until transmission is complete, think it does it using a timer interrupt) but the sheer simplicity of it is remarkable (sketches upon request) I don't know for what they were designed, but i tried them out and for me the 1-way communication, and the total lack of security meant that i decided that i would not use them for anything other then controlling kids-toys
For my application, where 3 times an hour 5 values and a character need to be transmitted this I thought would be good enough.
This discussion starting now wakes my interest in further developing this transmission protocol, ie ASK protocol and VW or Radiohead.
Title: Re: Serial Input Basics: example 5, question
Post by: Deva_Rishi on Jan 22, 2019, 09:33 pm
For my application, where 3 times an hour 5 values and a character need to be transmitted this I thought would be good enough.
Yeah it probably is.. a bit unfortunate that we went such a long way, to look for a mistake in a wheel (which there isn't really Robin's code is good) while there is a part (VW) that would remove any errors in the inputstream. (I suppose we would still be left with the strange strtok() behavior, which if it persists, a bug report to the ESP-core people should be filed.)