Problem with parsing big packets on Serial port Arduino Nano

Really nobody can advise? Maybe this is a problem with the Stream class. Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

Gennady: Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

As a piece of English that makes no sense.

And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

If you have a buffer that is bigger than the message that is being sent then I can think of no technical reason why the message cannot be parsed. However I know nothing about the UBLOX library you are using and that is where I would start searching for the problem.

Please post a link to the datasheet with the specification for the data that is SENT by the GPS device.

An example of a typical message would also be a big help. If it contains non-printing characters be sure to identify them all. They can be the key to effective parsing.

...R

Thanks for the answer! Oddly enough, the Serial library at compilation was taken from 163. I deleted 163. This did not help.

Today I experimented with the messages in the package. NEO 57600baud, 5Hz. The working version is the following and the only one.

UBX NAV-SOL, Size 60, 'Navigation Solution' UBX NAV-PVT, Size 100, 'Navigation PVT Solution' UBX NAV-POSLLH, Size 36, 'Geodetic Position' UBX NAV-VELNED, Size 44, 'Velocity in WGS 84'

Total 240 bytes.

In this order, NEO builds them.

I add MON-HW2 long 36 byte. He stands in front of SOL on some time lead. But its frequency of appearance is 1 Hz. At this point, the parsing does not work. After that, I removed MON-HW2, but I inserted the message DOP 26 byte. It fell AFTER the POSLLH message. With him, the parsing also disappeared. At lower speeds 38400, 19200 the same.

I noticed that after pressing the reset button, the first parsing takes place under any conditions. It seems that with a packet of more than 240 bytes, some kind of buffer overflows.

Maybe problem in Stream.h lib?

I do not know what to do anymore ((((

https://www.u-blox.com/sites/default/files/products/documents/u-blox8-M8_ReceiverDescrProtSpec_%28UBX-13003221%29_Public.pdf

From a quick look at Section 32 of the linked PDF it seems that a message has the style $GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

...R

Robin2: From a quick look at Section 32 of the linked PDF it seems that a message has the style $GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

...R

No! It NMEA protocol. I use only UBX! Message some like this (up to 100byte and next Message start w/o any CR/LF :

//UBX-NAV-VELNED

16:11:32 0000 B5 62 01 12 24 00 00 8B 39 1D 07 00 00 00 EC FF µb $ 9 ìÿ 0010 FF FF DF FF FF FF 27 00 00 00 15 00 00 00 71 C2 ÿÿßÿÿÿ' q 0020 D1 01 83 00 00 00 3C 01 47 00 2C 60 Ñ < G ,`

//UBX-NAV-DOP 16:11:33 0000 B5 62 01 02 1C 00 90 8C 39 1D 53 96 4D 12 96 DF µb 9 S M ß 0010 05 1E 9B FC 01 00 0A 99 01 00 36 15 00 00 13 1D ü 6 0020 00 00 28 3D (=

total packet up to 300byte and more.

Robin2: And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

Deva_Rishi: I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

I think I spent an hour with Google trying to figure that out and then the simple solution occurred to me - took all of 5 minutes.

...R

Gennady: No! It NMEA protocol. I use only UBX!

What section of the PDF is that in ?

Why do you seem to be using the more difficult system?

...R

Thank you for your attention to my problem! Below are links to pages with descriptions of individual messages. The format UBX is good, compact, belongs to Ublox, I use it because it is used in autopilot. NMEA it is very old format for ships. All the time I think that the problem is not in the program, but in the buffer or something else

33.17 UBX-NAV (0x01) ... 301

33.17.5 UBX-NAV-DOP (0x01 0x04) ........................................................................................ 304

33.17.13 UBX-NAV-POSLLH (0x01 0x02).................................................................................... 312 33.17.14 UBX-NAV-PVT (0x01 0x07) ........................................................................................ 313 33.17.20 UBX-NAV-SOL (0x01 0x06) ........................................................................................ 322 33.17.31 UBX-NAV-VELNED (0x01 0x12) .................................................................................. 337

I gave the parsing code above. As for me, everything is right there. I read somewhere that Serial uses the Stream lib and the problem with receiving big data in the restrictions in Stream. "Some of the libraries that rely on Stream include : Serial..." Or the problem may be that the Serial lib buffer in the CPU RAM is full as a result of the program execution. And new data does not come. This can be confirmed by the fact that after pressing the reset button ALWAYS ONE parsing occurs. And then they are not, or there is very rarely, once a minute. This is with a packet of more than 240 bytes !!! Can a program in a loop have to clear buffer in the CPU RAM by a command Serialend () or another command? I do not know how well the Serial library is written for this case.

The UBX frame structure is described on pages 134/5

With that type of message you need code to detect the start characters and then to interpret the length of the payload.

I don't know that I can give you any more assistance.

...R

I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions…

  1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

  1. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

  2. Is there something between the packets like carriage-return putting it into that stable state?

  3. Is it overruning any buffer and writing received characters to places it shouldn’t?

I’ve never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

Robin2: With that type of message you need code to detect the start characters and then to interpret the length of the payload.

...R

p.313 UBX-NAV-PVT Header Class ID Length (Bytes) Payload Checksum Message Structure 0xB5 0x62 0x01 0x07 92 see below CK_A CK_B

From lib UBLOX.h:

const uint8_t _ubxHeader[2] = {0xB5, 0x62};
    const uint8_t _ubxNavPvt_msgClass = 0x01;
    const uint8_t _ubxNavPvt_msgId = 0x07;
    const uint16_t _ubxNavPvt_msgLen = 96;//total data 92+header 2+ mess class/type 2=96

Parsing From lib UBLOX.cpp::

/* parse the uBlox data */
bool UBLOX::_parse(uint8_t msg_class,uint8_t msg_id,uint16_t msg_length)
{
 // read a byte from the serial port
 while (_bus->available()) {
 _byte = _bus->read();
 // identify the packet header
 if (_parserState < 2) {
 if (_byte == _ubxHeader[_parserState]) {
 _parserState++;
 } else {
 _parserState = 0;
 }
 } else {
 if ((_parserState - 2) < msg_length) {
 *((uint8_t *) &_tempPacket + _parserState - 2) = _byte;
 }
 _parserState++;
 // compute checksum
 if ((_parserState - 2) == msg_length) {
 _calcChecksum(_checksum,((uint8_t *) &_tempPacket),msg_length);
 } else if ((_parserState - 2) == (msg_length + 1)) {
 if (_byte != _checksum[0]) {
 _parserState = 0;
 }
 } else if ((_parserState - 2) == (msg_length + 2)) {
 _parserState = 0;
 if (_byte == _checksum[1]) {
 memcpy(&_validPacket,&_tempPacket,sizeof(_validPacket));
 return true;
 }
 } else if (_parserState > (msg_length + 4) ) {
 _parserState = 0;
 }
 }
 }
 return false;
}

/* uBlox checksum */
void UBLOX::_calcChecksum(uint8_t* CK, uint8_t* payload, uint16_t length)
{
 CK[0] = 0;
  CK[1] = 0;
 for (uint8_t i = 0; i < length; i++) {
 CK[0] += payload[i];
 CK[1] += CK[0];
 }
}

From you, I want to get advice on the work of the Stream and Serial libraries from the Arduino core. The problem is in them !!!!! This is indicated by the fact that after a reset, the first time parsing is ALWAYS correct once. Then not or rarely! The buffer of the Serial library (filled independently of the program cycle through ISR UART) is overflowed with data and new data AT ALL do not arrive. How to clear this buffer? Or a problem in Stream with a large amount of data. Here I need your great experience! Thanks in a

MorganS: I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions...

  1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

  1. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

  2. Is there something between the packets like carriage-return putting it into that stable state?

  3. Is it overruning any buffer and writing received characters to places it shouldn't?

I've never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

  1. GPS transmits packets continuously. The length of the packet is up to 100 millis, I have 200 millis repetition time (5 Hz). Necessary messages are placed in the package even at a speed of 19200. Leave free time. I need to get only one of the five messages in the package. Separate messages are joined in a packet without intervals. There is no command to transfer, the transfer goes automatically and continuously. 2.

I think the parser works that way. Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck, I think, it does not receive data from the Serial library at all, or it receives very rarely - once a minute.

  1. No, first byte of next messages come.

4.No

Sample of packet with two messages: 12:54:28 R -> UBX NAV-DOP, Size 26, 'Dilution of Precision' 12:54:28 R -> UBX NAV-VELNED, Size 44, 'Velocity in WGS 84'

b5 62 01 04 12 00 f8 f1 ad 21 c0 00 a7 00 5e 00 µb....шс­!А.§.^. 85 00 64 00 4b 00 43 00 0a 34 b5 62 01 12 24 00 ….d.K.C..4µb..$. f8 f1 ad 21 07 00 00 00 05 00 00 00 fa ff ff ff шс­!........ъяяя 0b 00 00 00 09 00 00 00 7a 26 25 00 64 00 00 00 ........z&%.d... f2 84 2d 00 d1 ec

If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon.

Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck

Which is it?

MorganS: If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon. Which is it?

If the packet is larger than 240 bytes, the parsing does not work or it works very rarely, about once a minute. I checked the UBLOX library along with the author. He developed it for T_3.5 / 3.6, everything works for him. Please explain how the Serial library RX buffer works. What happens if it is full, how to clean it?

If the Serial buffer is full then you have screwed up. The simple way to do that would be to put delay(100) in your program.

I actually do that in the setup() of some of my important projects. There are delays initializing the LCD screen and other peripherals. So I clear the Serial buffer at the end of setup() like this...

  while(Serial.available()) Serial.read();

MorganS: If the Serial buffer is full then you have screwed up. The simple way to do that would be to put delay(100) in your program.

I actually do that in the setup() of some of my important projects. There are delays initializing the LCD screen and other peripherals. So I clear the Serial buffer at the end of setup() like this...

  while(Serial.available()) Serial.read();

Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough. If I understand correctly, then the buffer is filled with data by an interrupt from the UART RX. Therefore, it is necessary to make room for new data before starting the parsing in LOOP ().There should be a clear buffer command. After all, this is just zeroing the variable of the buffer filling counter. Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first packet, as after resetting the CPU. Of course, I will try also your buffer cleaning solution. Thanks!

Gennady: Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first packet, as after resetting the CPU.

Yes it does, also you can set the pointer to the last byte = pointer to the first byte (Serial.begin() sets both to 0, from the (ring)buffer offset. I think, i read it somewhere but i haven't checked inside hwSerial.cpp, or maybe i did, but haven't just now to verify my memory. You should, you could also add the function to clear the buffer to hwSerial as you think it suits you most. If the GPS is constantly broadcasting, there should be a break of a certain length in between broadcasts (or at least one would hope so) and you can look for that before you start listening. Since the main part is raw data, it may be hard to look for a header.

Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough.

in some way yes, though with you baudrate that should not really be an issue. Either way check out this way of receiving DMX on an Arduino, with having the ISR vectors redefined and the ISR's in the Sketch. There is a more modern way using the Conceptinetics DMX library, you can get some more details out of that.

Gennady:
Sooner or later, any buffer can be filled with data. It is not always possible to do the parsing often enough. Maybe just the CPU performance is not enough.

No. Serial is really slow. Even the slowest Arduino can perform thousands of operations between each Serial character arriving.

If you are burning up those thousands of instructions doing floating-point calculations then that doesn’t mean Serial is broken. It means you or your library are doing something wrong.

If I understand correctly, then the buffer is filled with data by an interrupt from the UART RX. Therefore, it is necessary to make room for new data before starting the parsing in LOOP ().

Nope. Parsing by using Serial.read() is the only thing that should be taking characters out of that buffer. If parsing is working fast enough then it will empty the buffer before the next packet finishes transmission.

There should be a clear buffer command. After all, this is just zeroing the variable of the buffer filling counter. Can Serial.End () and Serial.began () apply and then enable parsing? Then the parsing will checknew data like in first packet, as after resetting the CPU.

Double nope! If you erase the buffer then you would have to wait at least 100 milliseconds for the next packet to start. Then wait for that packet to finish. That is a long time for the Arduino to waste when it already had the data buffered and waiting for the parser.

Of course, I will try also your buffer cleaning solution. Thanks!

No. Don’t use that code. Just verify that the parser always completes its job before the next packet is complete in the Serial buffer.

Print out Serial.available() after getting that first packet. It should be a small number like 0 or maybe 100.

Thank! I will use your advice and conduct experiments.