Go Down

Topic: Serial Input Basics: example 5, question (Read 2083 times) previous topic - next topic

Deva_Rishi

One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
Second if the startmarker ends up in the middle of the message (x+1), something is really wrong ! maybe the buffer was not completely empty when the new message was sent.
I don't see  a problem with the residual bytes in x+2, they are behind a null terminator.
Quote
My thinking has been that either the strtok implementation in the esp8266 core has a problem, or, more likely in my opinion, that the NULL is somehow missing. Perhaps because the total received chars overruns the 32 character boundary.
both are possible, but since we are not low on memory at the moment how about increasing the buffer to something that is easily enough(100 bytes ?) until we can figure out what is going on.
To 'Correct' you have to be Correct. (and not be condescending..)

Deva_Rishi

To prevent any fog clouding the issue at hand: my post #34 contains the bare-bones program with all essentials related to reception and parsing available. No interrupts are disabled anywhere, not in the original, not in the current bare-bones program.
but not within </> code tags. that is why i missed it.
what is 'wire' being used for ?
also i think the corruption of the data may have something to do with
Code: [Select]
if (ndx >= numChars) {
          ndx = numChars - 1;
        }
which prevents writing beyond the size of the array, but every new character will just overwrite the previous one. The test is valid, but the error handling is insufficient.
To 'Correct' you have to be Correct. (and not be condescending..)

brice3010

One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
But it currently results in a order of magnitude reduction in crashes.

Second if the startmarker ends up in the middle of the message (x+1), something is really wrong ! maybe the buffer was not completely empty when the new message was sent.
How to find out?

I don't see  a problem with the residual bytes in x+2, they are behind a null terminator. both are possible, but since we are not low on memory at the moment how about increasing the buffer to something that is easily enough(100 bytes ?) until we can figure out what is going on.
But why did they get there?

These residual bytes 41 and 1 are still there after about 4320 runs at this moment.

Should I restart with a buffer increased from 32 to your suggested 100?

brice3010

but not within </> code tags. that is why i missed it.
what is 'wire' being used for ?
The wire.h is used in the original program to accomodate use of I2C

also i think the corruption of the data may have something to do with
Code: [Select]
if (ndx >= numChars) {
          ndx = numChars - 1;
        }

which prevents writing beyond the size of the array, but every new character will just overwrite the previous one. The test is valid, but the error handling is insufficient.
How would you suggest to improve this?

cattledog

#64
Jan 19, 2019, 07:58 am Last Edit: Jan 19, 2019, 08:02 am by cattledog
Quote
One thing i understood about Robin's system with a startmarker, an endmarker and a separator is that none of those can be part of the actual data, a '.' does not meet those criteria.
The '.' is not used as an end marker. The start marker is '<' and the end marker is '>' . The '.' is a separator. The way you manage strtok, it is not going to be a separator for its earlier appearances in the data of the array.

Quote
These residual bytes 41 and 1 are still there after about 4320 runs at this moment.
I'm not sure how they got there, but there is nothing to clear the buffer, and since the buffer of 32 is larger than the received data those characters will just sit there. You can clear the buffer with this

Code: [Select]
if (newData == true) {
    memset(tempChars, '\0', sizeof(tempChars));
    strcpy(tempChars, receivedChars);
    //rest of code


Quote
Should I restart with a buffer increased from 32 to your suggested 100?
I think this a good idea to help see if there are rare cases where the received characters over run the 32 character buffer and the final separator gets overwritten.

brice3010

...

I'm not sure how they got there, but there is nothing to clear the buffer, and since the buffer of 32 is larger than the received data those characters will just sit there. You can clear the buffer with this

Code: [Select]
if (newData == true) {
    memset(tempChars, '\0', sizeof(tempChars));
    strcpy(tempChars, receivedChars);
    //rest of code


I think this a good idea to help see if there are rare cases where the received characters over run the 32 character buffer and the final separator gets overwritten.
Done. Buffer increased from 32 to 64, to keep the monitor screen display within manageable size. Downloaded and restarted. I insert a yield(); command in the Serial.print output to prevent, at 1200 baud, the ESP8266 crashing on watchdog timer.

I will let this run for a day or so. The individual transmissions still take place with exactly 5 second spacings between D, A and B, and 10 seconds between B and D (C currently not used).

Question 1: the appearance of < in the message x+1 still puzzles me. Is there any possible explanation?

Question 2: and the code below, is its purpose to reset the ndx counter in case no endmarker is received by the end of the 32 byte buffer? And if that is the case then the else after that if is not executed?
I refer to message #61 from Deva_Rishi.

Code: [Select]

      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }

cattledog

Quote
Question 1: the appearance of < in the message x+1 still puzzles me. Is there any possible explanation?
Not that I have been able to come up with other than overlapping messages which you say can't occur.

Quote
Question 2: and the code below, is its purpose to reset the ndx counter in case no endmarker is received by the end of the 32 byte buffer?
It does not reset the ndx counter, but holds it at its max value so that no characters are written to a memory location outside of the array range.

Quote
And if that is the case then the else after that if is not executed?
No, the else condition is executed when the end marker is received.


Robin2

These residual bytes 41 and 1 are still there after about 4320 runs at this moment.
When a new message arrives which is shorter than the previous message the tail of the previous data can remain in the buffer. Normally that is never noticeable because the cstring functions never look past the terminating NULL from the current message - in other words the first NULL that is detected.

You could adapt the code to specifically write space characters into every place in the buffer before starting to receive a new message, but I have never found it necessary.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

brice3010

During the 6 hours the test currently runs one crash occured:

actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .

ASCII message: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Parsing crashed at the fifth character which is not present in the actual message

During this run there never was anything written beyond the 23 or 24 bytes being expected.

Clearly this one crash is due to a corrupted byte received.

Maybe some kind of message integrity check needs to be included in the sent message? If so, how best to program?


PS: a joke I just heard from my wife: "The oldest computer was an Apple, owned by Adam and Eve. It had very limited memory, just one bite and it crashed".

Robin2

#69
Jan 19, 2019, 02:38 pm Last Edit: Jan 19, 2019, 02:39 pm by Robin2
During the 6 hours the test currently runs one crash occured:

actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .

ASCII message: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Parsing crashed at the fifth character which is not present in the actual message
What do you mean by the 5th character? I reckon it is the 255 (which is the 8th character) that is causing the problem

A very simple integrity check is to XOR all the bytes and include the XOR value as the last byte. However that would mean that your system would have to be able to handle any value from 0 to 255 so that the use of < and > as start and end markers would not work. Maybe the simplest solution is to Serial.print() the XOR value with suitable padding to ensure it is always 3 bytes long.

Remind us what is sending the data and what baud rate is being used?

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

brice3010

What do you mean by the 5th character? I reckon it is the 255 (which is the 8th character) that is causing the problem

A very simple integrity check is to XOR all the bytes and include the XOR value as the last byte. However that would mean that your system would have to be able to handle any value from 0 to 255 so that the use of < and > as start and end markers would not work. Maybe the simplest solution is to Serial.print() the XOR value with suitable padding to ensure it is always 3 bytes long.

Remind us what is sending the data and what baud rate is being used?

...R
The baud rate used is 1200: this allows the maximum possible range for the HC12 radio.

The transmitted message is composed as follows:

Code: [Select]


    Serial.print('<'); // this section for HC-12 transmission
    Serial.print(cycleValue);  // ref Robin2's Serial Output Basics
    Serial.print(',');   // http://forum.arduino.cc/index.php?topic=396450.0
    //    Serial.print(temperatureT1); // send temperature read from analog input temperaturePin
    Serial.print(get3231Temp()); // read DS3231 temperature
    Serial.print(',');
    Serial.print(batteryVoltage); // send battery voltage
    Serial.print(',');
    Serial.print(now.minute(), DEC); // send minute
    Serial.print(',');
    Serial.print(now.second(), DEC); // send hour
    Serial.print(",A.>"); // send sensor identifier
    Serial.println();


where cycleValue, temperatureT1, batteryVoltage are floats, and now.minute() and now.second() are integers derived from the RTClibExtended.h library.

Sorry my mistake for stating the 5th character: you are correct.

Serial.print the XOR value : how should I include this in the transmitted serial.print?
Thanks!!


Robin2

The baud rate used is 1200: this allows the maximum possible range for the HC12 radio.

Serial.print the XOR value : how should I include this in the transmitted serial.print?

Just put it in before the closing >

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

brice3010

Just put it in before the closing >

...R
Can you please help writing this XOR calculation and writing?

cattledog

Quote
actual message: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .
Quote
Parsing crashed at the fifth eight character which is not present in the actual message

During this run there never was anything written beyond the 23 or 24 bytes being expected.

Clearly this one crash is due to a corrupted byte received.
Before getting into error checking and checksums, I want to understand a bit more about what is going on.

Please describe more about the "crash". Did the ESP freeze, reset, ?

There is a lot more wrong with this data than an incorrect character. There are a missing 9 or 10 bytes. A simple test for message length would have picked this up. It looks like thee was a failure of the temperature and voltage readings. Perhaps they can be tested for correctness when measured.

Is this the first time that the print out has not reached "5" and the crash occurs at the last parsing?

I will test, but I think that strtok should have handled the mangled string without crashing on a UNO and just reported strange or NULL values.

The behaviour of strtok in the esp8266 environment continues to interest me.

brice3010

#74
Jan 19, 2019, 06:54 pm Last Edit: Jan 19, 2019, 06:58 pm by brice3010
Before getting into error checking and checksums, I want to understand a bit more about what is going on.

Please describe more about the "crash". Did the ESP freeze, reset, ?
The parsing index went to 4, meaning that when parsing the 4th value (the first integer after 3 floats) it went good, and the crash occured at next value, the second integer:

actual data: 4 1 . 6 7 , 2 ⸮ 0 , 2 , A .
ASCII data: 52 49 46 54 55 44 50 255 48 44 50 44 65 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 1  2  3  4

Then "exception 28" occured, a stack dump showed up, and the ESP8266 automatically restarted and continued.


There is a lot more wrong with this data than an incorrect character. There are a missing 9 or 10 bytes. A simple test for message length would have picked this up. It looks like thee was a failure of the temperature and voltage readings. Perhaps they can be tested for correctness when measured.
Interesting. What I will do is limit to one transmitter and connect that too to a serial monitor and readout whatever is being transmitted.

Is this the first time that the print out has not reached "5" and the crash occurs at the last parsing?
Yes, at least on this run (10hrs so far with this one crash).

I will test, but I think that strtok should have handled the mangled string without crashing on a UNO and just reported strange or NULL values.

The behaviour of strtok in the esp8266 environment continues to interest me.
That pleases me; the ESP8266 has so much potential it would be nice to get this sorted out.

Go Up