Problem with parsing big packets on Serial port Arduino Nano

[MorganS]

I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions...

1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

2. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

3. Is there something between the packets like carriage-return putting it into that stable state?

4. Is it overruning any buffer and writing received characters to places it shouldn't?

I've never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

[Gennady]

With that type of message you need code to detect the start characters and then to interpret the length of the payload.

...R

p.313 UBX-NAV-PVT
                                  Header      Class ID      Length (Bytes)    Payload      Checksum
Message Structure     0xB5 0x62 0x01 0x07    92                     see below   CK_A CK_B

From lib UBLOX.h:

Code: [Select]


const uint8_t _ubxHeader[2] = {0xB5, 0x62};
    const uint8_t _ubxNavPvt_msgClass = 0x01;
    const uint8_t _ubxNavPvt_msgId = 0x07;
    const uint16_t _ubxNavPvt_msgLen = 96;//total data 92+header 2+ mess class/type 2=96

 

Parsing From lib UBLOX.cpp::

Code: [Select]




/* parse the uBlox data */
bool UBLOX::_parse(uint8_t msg_class,uint8_t msg_id,uint16_t msg_length)
{
 // read a byte from the serial port
 while (_bus->available()) {
 _byte = _bus->read();
 // identify the packet header
 if (_parserState < 2) {
 if (_byte == _ubxHeader[_parserState]) {
 _parserState++;
 } else {
 _parserState = 0;
 }
 } else {
 if ((_parserState - 2) < msg_length) {
 *((uint8_t *) &_tempPacket + _parserState - 2) = _byte;
 }
 _parserState++;
 // compute checksum
 if ((_parserState - 2) == msg_length) {
 _calcChecksum(_checksum,((uint8_t *) &_tempPacket),msg_length);
 } else if ((_parserState - 2) == (msg_length + 1)) {
 if (_byte != _checksum[0]) {
 _parserState = 0;
 }
 } else if ((_parserState - 2) == (msg_length + 2)) {
 _parserState = 0;
 if (_byte == _checksum[1]) {
 memcpy(&_validPacket,&_tempPacket,sizeof(_validPacket));
 return true;
 }
 } else if (_parserState > (msg_length + 4) ) {
 _parserState = 0;
 }
 }
 }
 return false;
}

/* uBlox checksum */
void UBLOX::_calcChecksum(uint8_t* CK, uint8_t* payload, uint16_t length)
{
 CK[0] = 0;
  CK[1] = 0;
 for (uint8_t i = 0; i < length; i++) {
 CK[0] += payload[i];
 CK[1] += CK[0];
 }
}





From you, I want to get advice on the work of the Stream and Serial libraries from the Arduino core. The problem is in them !!!!! This is indicated by the fact that after a reset, the first time parsing is ALWAYS correct once. Then not or rarely! The buffer of the Serial library
(filled independently of the program cycle through ISR UART) is overflowed with data and new data AT ALL do not arrive. How to clear this buffer? Or a problem in Stream with a large amount of data. Here I need your great experience!
Thanks in a

[Gennady]

I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions...

1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

2. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

3. Is there something between the packets like carriage-return putting it into that stable state?

4. Is it overruning any buffer and writing received characters to places it shouldn't?

I've never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

1. GPS transmits packets continuously. The length of the packet is up to 100 millis, I have 200 millis repetition time (5 Hz). Necessary messages are placed in the package even at a speed of 19200. Leave free time. I need to get only one of the five messages in the package. Separate messages are joined in a packet without intervals. There is no command to transfer, the transfer goes automatically and continuously.
2.

I think the parser works that way. Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck, I think, it does not receive data from the Serial library at all, or it receives very rarely - once a minute.

3. No, first byte of next  messages come.

4.No

Sample of packet with two messages:
12:54:28  R -> UBX NAV-DOP,  Size  26,  'Dilution of Precision'
12:54:28  R -> UBX NAV-VELNED,  Size  44,  'Velocity in WGS 84'


b5 62 01 04 12 00 f8 f1 ad 21 c0 00 a7 00 5e 00  µb....шс­!А.§.^.
85 00 64 00 4b 00 43 00 0a 34 b5 62 01 12 24 00  ….d.K.C..4µb..$.
f8 f1 ad 21 07 00 00 00 05 00 00 00 fa ff ff ff  шс­!........ъяяя
0b 00 00 00 09 00 00 00 7a 26 25 00 64 00 00 00  ........z&%.d...
f2 84 2d 00 d1 ec

[MorganS]

If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon.

Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck

Which is it?

[Gennady]

If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon.
Which is it?

If the packet is larger than 240 bytes, the parsing does not work or it works very rarely, about once a minute. I checked the UBLOX library along with the author. He developed it for T_3.5 / 3.6, everything works for him. Please explain how the Serial library RX buffer works. What happens if it is full, how to clean it?

[MorganS]

If the Serial buffer is full then you have screwed up. The simple way to do that would be to put delay(100) in your program.

I actually do that in the setup() of some of my important projects. There are delays initializing the LCD screen and other peripherals. So I clear the Serial buffer at the end of setup() like this...

Code: [Select]

  while(Serial.available()) Serial.read();

[Gennady]

If the Serial buffer is full then you have screwed up. The simple way to do that would be to put delay(100) in your program.

I actually do that in the setup() of some of my important projects. There are delays initializing the LCD screen and other peripherals. So I clear the Serial buffer at the end of setup() like this...

Code: [Select]

  while(Serial.available()) Serial.read();

Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough.
If I understand correctly, then the buffer is filled with data by an interrupt from the UART RX. Therefore, it is necessary to make room for new data before starting the parsing in LOOP ().There should be a clear buffer command. After all, this is just zeroing the variable of the buffer filling counter. Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first  packet, as after resetting the CPU.
Of course, I will try also your buffer cleaning solution. Thanks!

[Deva_Rishi]

Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first  packet, as after resetting the CPU.

Yes it does, also you can set the pointer to the last byte = pointer to the first byte (Serial.begin() sets both to 0, from the (ring)buffer offset. I think, i read it somewhere but i haven't checked inside hwSerial.cpp, or maybe i did, but haven't just now to verify my memory. You should, you could also add the function to clear the buffer to hwSerial as you think it suits you most.
If the GPS is constantly broadcasting, there should be a break of a certain length in between broadcasts (or at least one would hope so) and you can look for that before you start listening. Since the main part is raw data, it may be hard to look for a header.

Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough.

in some way yes, though with you baudrate that should not really be an issue. Either way check out this way of receiving DMX on an Arduino, with having the ISR vectors redefined and the ISR's in the Sketch. There is a more modern way using the Conceptinetics DMX library, you can get some more details out of that.

[MorganS]

Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough.

No. Serial is really slow. Even the slowest Arduino can perform thousands of operations between each Serial character arriving.

If you are burning up those thousands of instructions doing floating-point calculations then that doesn't mean Serial is broken. It means you or your library are doing something wrong.

If I understand correctly, then the buffer is filled with data by an interrupt from the UART RX. Therefore, it is necessary to make room for new data before starting the parsing in LOOP ().

Nope. Parsing by using Serial.read() is the only thing that should be taking characters out of that buffer. If parsing is working fast enough then it will empty the buffer before the next packet finishes transmission.

There should be a clear buffer command. After all, this is just zeroing the variable of the buffer filling counter. Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first  packet, as after resetting the CPU.

Double nope! If you erase the buffer then you would have to wait at least 100 milliseconds for the next packet to start. Then wait for that packet to finish. That is a long time for the Arduino to waste when it already had the data buffered and waiting for the parser.

Of course, I will try also your buffer cleaning solution. Thanks!

No. Don't use that code. Just verify that the parser always completes its job before the next packet is complete in the Serial buffer.

Print out Serial.available() after getting that first packet. It should be a small number like 0 or maybe 100.

[Gennady]

Thank! I will use your advice and conduct experiments.