Problem with parsing big packets on Serial port Arduino Nano

Hi!

IDE 1.8.2, Nano, parsing lib https://github.com/bolderflight/UBLOX/tree/master/src

When parsing data coming from a GPS NEO M8N 200mS update, there are constant failures or data in general do not pass parsing at a packet size of more than 240 bytes (at speeds of 19200, 38400, 57600).I need 290 bytes! At other speeds I did not check, since they do not suit me. I tested with the revised HardwareSerial.h file with the RX buffer increased to 330 bytes.
Compiling a sketch gives: Global variables use 802 bytes (39%) of dynamic memory, leaving 1246 bytes for local variables. Maximum is 2048 bytes.

#if !defined(SERIAL_TX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64 //64
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 330 //16
#else
#define SERIAL_RX_BUFFER_SIZE 330 //64
#endif
#endif

The program in the cycle consists of two parts. The first part is the parsing of the 100 byte portion of the packet and the writing of values ​​into variables. After that, a trigger is set up that allows the operation of another part - my code.
My code does some transformations with the variables obtained above and produces the result on the UART TX. After that, the trigger is reset. Thus, my code is executed only after receiving new data through library.

loop{

if (gps.readSensor())
    { digitalWrite(6, HIGH);
      Longitude_deg = gps.getLongitude_deg();
      Latitude_deg  = gps.getLatitude_deg();   
      GnssFixOk     = gps.isGnssFixOk();
      startLatitude_rad  = gps.getLatitude_rad();
      startLongitude_rad = gps.getLongitude_rad();
      MSLHeight_m   = gps.getMSLHeight_m();
      NumSatellites = gps.getNumSatellites();
      triggerParse = 1;
      digitalWrite(6, LOW);
    }

if (  triggerParse == 1 ) 
    { digitalWrite(9, HIGH);

//my code here....

      triggerParse = 0;
      digitalWrite(9, LOW);

    }
}

Please advise what I can try to do?

I also give a picture for the case when everything works with a packet length of 240 bytes- channels from top to bottom: parsing work, UART RX, my code work, UART TX.

Thanks in advance!

Gennady

Please post a complete program.

What is the purpose (or intended purpose) of the stuff in the first code block in your Original Post?

...R

what type of variable do you use for parsing ? In fact yes post the complete sketch or at least the parsing functions...

I tested with the revised file from //hardware/avr/arduino/ HardwareSerial.h with the RX buffer increased to 330 bytes:

#if !defined(SERIAL_TX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64 //64
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 330 //16
#else
#define SERIAL_RX_BUFFER_SIZE 330 //64
#endif
#endif

sketch

#include  "UBLOX.h"
#define DEBUG
#define UART_SPEED       57600

double   startLatitude_rad       = 0;
double   startLongitude_rad      = 0;
bool     GnssFixOk          = false;
double   Latitude_deg       = 0;
double   Longitude_deg      = 0;
double   Latitude_rad       = 0;
double   Longitude_rad      = 0;
double   MSLHeight_m        = 0;
uint8_t  NumSatellites      = 0;
double   GroundSpeed_ms     = 0;
double   pointLatitude_rad  = 0;
double   pointLongitude_rad = 0;

byte   triggerParse = 0;

UBLOX gps(Serial, UART_SPEED);


void setup()
{

  pinMode(6, OUTPUT);
  pinMode(7, OUTPUT);
  pinMode(8, OUTPUT);
  pinMode(9, OUTPUT);

  gps.begin();

#ifdef DEBUG
  Serial.println("START ");
  Serial.println("-------------------------- ");
#endif



}

void loop(){

  if (gps.readSensor())
  { digitalWrite(6, HIGH);
    Longitude_deg = gps.getLongitude_deg();
    Latitude_deg  = gps.getLatitude_deg();
    GnssFixOk     = gps.isGnssFixOk();
    startLatitude_rad  = gps.getLatitude_rad();
    startLongitude_rad = gps.getLongitude_rad();
    MSLHeight_m   = gps.getMSLHeight_m();
    NumSatellites = gps.getNumSatellites();
    triggerParse = 1;
    digitalWrite(6, LOW);
  }

  if (  triggerParse == 1 )
  { digitalWrite(9, HIGH);

#ifdef DEBUG
    Serial.print("N ");    Serial.print(NumSatellites);
    Serial.print(" F ");  Serial.print(GnssFixOk);
    Serial.print(" Lat ");  Serial.print(Latitude_deg, 7);
    Serial.print(" Lon ");  Serial.print(Longitude_deg, 7);
    Serial.print(" Spd "); Serial.print(GroundSpeed_ms);
#endif

    triggerParse = 0;
    digitalWrite(9, LOW);

  }
}

Please advise what I can try to do?

Deva_Rishi:
what type of variable do you use for parsing ? In fact yes post the complete sketch or at least the parsing functions…

in UART sending only bytes. Parsing also byte per byte- pasing function from .cpp file from lib:

/* reads packets from the uBlox receiver */
bool UBLOX::readSensor()
{
	if (_parse(_ubxNavPvt_msgClass,_ubxNavPvt_msgId,_ubxNavPvt_msgLen)) {
		return true;
	} else {
		return false;
	}
}

/* parse the uBlox data */
bool UBLOX::_parse(uint8_t msg_class,uint8_t msg_id,uint16_t msg_length)
{
	// read a byte from the serial port
	while (_bus->available()) {
		_byte = _bus->read();
		// identify the packet header
		if (_parserState < 2) {
			if (_byte == _ubxHeader[_parserState]) {
				_parserState++;
			} else {
				_parserState = 0;
			}
		} else {
			if ((_parserState - 2) < msg_length) {
				*((uint8_t *) &_tempPacket + _parserState - 2) = _byte;
			}
			_parserState++;
			// compute checksum
			if ((_parserState - 2) == msg_length) {
				_calcChecksum(_checksum,((uint8_t *) &_tempPacket),msg_length);
			} else if ((_parserState - 2) == (msg_length + 1)) {
				if (_byte != _checksum[0]) {
					_parserState = 0;
				}
			} else if ((_parserState - 2) == (msg_length + 2)) {
				_parserState = 0;
				if (_byte == _checksum[1]) {
					memcpy(&_validPacket,&_tempPacket,sizeof(_validPacket));
					return true;
				}
			} else if (_parserState > (msg_length + 4) ) {
				_parserState = 0;
			}
		}
	}
	return false;
}

/* uBlox checksum */
void UBLOX::_calcChecksum(uint8_t* CK, uint8_t* payload, uint16_t length)
{
	CK[0] = 0;
  CK[1] = 0;
	for (uint8_t i = 0; i < length; i++) {
		CK[0] += payload[i];
		CK[1] += CK[0];
	}
}

Really nobody can advise? Maybe this is a problem with the Stream class. Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

Gennady: Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

As a piece of English that makes no sense.

And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

If you have a buffer that is bigger than the message that is being sent then I can think of no technical reason why the message cannot be parsed. However I know nothing about the UBLOX library you are using and that is where I would start searching for the problem.

Please post a link to the datasheet with the specification for the data that is SENT by the GPS device.

An example of a typical message would also be a big help. If it contains non-printing characters be sure to identify them all. They can be the key to effective parsing.

...R

Thanks for the answer! Oddly enough, the Serial library at compilation was taken from 163. I deleted 163. This did not help.

Today I experimented with the messages in the package. NEO 57600baud, 5Hz. The working version is the following and the only one.

UBX NAV-SOL, Size 60, 'Navigation Solution' UBX NAV-PVT, Size 100, 'Navigation PVT Solution' UBX NAV-POSLLH, Size 36, 'Geodetic Position' UBX NAV-VELNED, Size 44, 'Velocity in WGS 84'

Total 240 bytes.

In this order, NEO builds them.

I add MON-HW2 long 36 byte. He stands in front of SOL on some time lead. But its frequency of appearance is 1 Hz. At this point, the parsing does not work. After that, I removed MON-HW2, but I inserted the message DOP 26 byte. It fell AFTER the POSLLH message. With him, the parsing also disappeared. At lower speeds 38400, 19200 the same.

I noticed that after pressing the reset button, the first parsing takes place under any conditions. It seems that with a packet of more than 240 bytes, some kind of buffer overflows.

Maybe problem in Stream.h lib?

I do not know what to do anymore ((((

https://www.u-blox.com/sites/default/files/products/documents/u-blox8-M8_ReceiverDescrProtSpec_%28UBX-13003221%29_Public.pdf

From a quick look at Section 32 of the linked PDF it seems that a message has the style $GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

...R

Robin2:
From a quick look at Section 32 of the linked PDF it seems that a message has the style
$GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E
and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

…R

No! It NMEA protocol. I use only UBX! Message some like this (up to 100byte and next Message start w/o any CR/LF :

//UBX-NAV-VELNED

16:11:32 0000 B5 62 01 12 24 00 00 8B 39 1D 07 00 00 00 EC FF µb $ 9 ìÿ
0010 FF FF DF FF FF FF 27 00 00 00 15 00 00 00 71 C2 ÿÿßÿÿÿ’ qÂ
0020 D1 01 83 00 00 00 3C 01 47 00 2C 60 Ñ < G ,`

//UBX-NAV-DOP
16:11:33 0000 B5 62 01 02 1C 00 90 8C 39 1D 53 96 4D 12 96 DF µb 9 S M ß
0010 05 1E 9B FC 01 00 0A 99 01 00 36 15 00 00 13 1D ü 6
0020 00 00 28 3D (=

total packet up to 300byte and more.

Robin2: And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

Deva_Rishi: I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

I think I spent an hour with Google trying to figure that out and then the simple solution occurred to me - took all of 5 minutes.

...R

Gennady: No! It NMEA protocol. I use only UBX!

What section of the PDF is that in ?

Why do you seem to be using the more difficult system?

...R

Thank you for your attention to my problem! Below are links to pages with descriptions of individual messages. The format UBX is good, compact, belongs to Ublox, I use it because it is used in autopilot. NMEA it is very old format for ships. All the time I think that the problem is not in the program, but in the buffer or something else

33.17 UBX-NAV (0x01) ... 301

33.17.5 UBX-NAV-DOP (0x01 0x04) ........................................................................................ 304

33.17.13 UBX-NAV-POSLLH (0x01 0x02).................................................................................... 312 33.17.14 UBX-NAV-PVT (0x01 0x07) ........................................................................................ 313 33.17.20 UBX-NAV-SOL (0x01 0x06) ........................................................................................ 322 33.17.31 UBX-NAV-VELNED (0x01 0x12) .................................................................................. 337

I gave the parsing code above. As for me, everything is right there. I read somewhere that Serial uses the Stream lib and the problem with receiving big data in the restrictions in Stream. "Some of the libraries that rely on Stream include : Serial..." Or the problem may be that the Serial lib buffer in the CPU RAM is full as a result of the program execution. And new data does not come. This can be confirmed by the fact that after pressing the reset button ALWAYS ONE parsing occurs. And then they are not, or there is very rarely, once a minute. This is with a packet of more than 240 bytes !!! Can a program in a loop have to clear buffer in the CPU RAM by a command Serialend () or another command? I do not know how well the Serial library is written for this case.

The UBX frame structure is described on pages 134/5

With that type of message you need code to detect the start characters and then to interpret the length of the payload.

I don't know that I can give you any more assistance.

...R

I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions…

  1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

  1. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

  2. Is there something between the packets like carriage-return putting it into that stable state?

  3. Is it overruning any buffer and writing received characters to places it shouldn’t?

I’ve never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

Robin2:
With that type of message you need code to detect the start characters and then to interpret the length of the payload.

…R

p.313 UBX-NAV-PVT
Header Class ID Length (Bytes) Payload Checksum
Message Structure 0xB5 0x62 0x01 0x07 92 see below CK_A CK_B

From lib UBLOX.h:

const uint8_t _ubxHeader[2] = {0xB5, 0x62};
    const uint8_t _ubxNavPvt_msgClass = 0x01;
    const uint8_t _ubxNavPvt_msgId = 0x07;
    const uint16_t _ubxNavPvt_msgLen = 96;//total data 92+header 2+ mess class/type 2=96

Parsing From lib UBLOX.cpp::

/* parse the uBlox data */
bool UBLOX::_parse(uint8_t msg_class,uint8_t msg_id,uint16_t msg_length)
{
 // read a byte from the serial port
 while (_bus->available()) {
 _byte = _bus->read();
 // identify the packet header
 if (_parserState < 2) {
 if (_byte == _ubxHeader[_parserState]) {
 _parserState++;
 } else {
 _parserState = 0;
 }
 } else {
 if ((_parserState - 2) < msg_length) {
 *((uint8_t *) &_tempPacket + _parserState - 2) = _byte;
 }
 _parserState++;
 // compute checksum
 if ((_parserState - 2) == msg_length) {
 _calcChecksum(_checksum,((uint8_t *) &_tempPacket),msg_length);
 } else if ((_parserState - 2) == (msg_length + 1)) {
 if (_byte != _checksum[0]) {
 _parserState = 0;
 }
 } else if ((_parserState - 2) == (msg_length + 2)) {
 _parserState = 0;
 if (_byte == _checksum[1]) {
 memcpy(&_validPacket,&_tempPacket,sizeof(_validPacket));
 return true;
 }
 } else if (_parserState > (msg_length + 4) ) {
 _parserState = 0;
 }
 }
 }
 return false;
}

/* uBlox checksum */
void UBLOX::_calcChecksum(uint8_t* CK, uint8_t* payload, uint16_t length)
{
 CK[0] = 0;
  CK[1] = 0;
 for (uint8_t i = 0; i < length; i++) {
 CK[0] += payload[i];
 CK[1] += CK[0];
 }
}

From you, I want to get advice on the work of the Stream and Serial libraries from the Arduino core. The problem is in them !!! This is indicated by the fact that after a reset, the first time parsing is ALWAYS correct once. Then not or rarely! The buffer of the Serial library
(filled independently of the program cycle through ISR UART) is overflowed with data and new data AT ALL do not arrive. How to clear this buffer? Or a problem in Stream with a large amount of data. Here I need your great experience!
Thanks in a

MorganS: I would start by modifying the library to print every character that it gets. Plus print the state. That would help answer the following questions...

  1. After the GPS has emitted its first valid packet, does it keep sending more valid packets?

It may need a command or acknowledgement to send the next one.

  1. Does the parser get stuck in a stable state? Stable is bad here. Any sequence of invalid characters should eventually get it back into the zero state that is looking for the start character.

  2. Is there something between the packets like carriage-return putting it into that stable state?

  3. Is it overruning any buffer and writing received characters to places it shouldn't?

I've never worked with that library but it does seem unlikely that it is so terrible that the parser gets stuck like that.

  1. GPS transmits packets continuously. The length of the packet is up to 100 millis, I have 200 millis repetition time (5 Hz). Necessary messages are placed in the package even at a speed of 19200. Leave free time. I need to get only one of the five messages in the package. Separate messages are joined in a packet without intervals. There is no command to transfer, the transfer goes automatically and continuously. 2.

I think the parser works that way. Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck, I think, it does not receive data from the Serial library at all, or it receives very rarely - once a minute.

  1. No, first byte of next messages come.

4.No

Sample of packet with two messages: 12:54:28 R -> UBX NAV-DOP, Size 26, 'Dilution of Precision' 12:54:28 R -> UBX NAV-VELNED, Size 44, 'Velocity in WGS 84'

b5 62 01 04 12 00 f8 f1 ad 21 c0 00 a7 00 5e 00 µb....шс­!А.§.^. 85 00 64 00 4b 00 43 00 0a 34 b5 62 01 12 24 00 ….d.K.C..4µb..$. f8 f1 ad 21 07 00 00 00 05 00 00 00 fa ff ff ff шс­!........ъяяя 0b 00 00 00 09 00 00 00 7a 26 25 00 64 00 00 00 ........z&%.d... f2 84 2d 00 d1 ec

If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon.

Therefore, with a packet length of up to 240 bytes, everything is fine! More than 240 bytes, it does not get stuck

Which is it?

MorganS: If there was a problem with Serial or Stream then millions of people would be having that problem.

If the problem is in the ublox library then maybe only a thousand people and they're not posting it here.

Check the characters that the library actually received and acted upon. Which is it?

If the packet is larger than 240 bytes, the parsing does not work or it works very rarely, about once a minute. I checked the UBLOX library along with the author. He developed it for T_3.5 / 3.6, everything works for him. Please explain how the Serial library RX buffer works. What happens if it is full, how to clean it?