Problem with parsing big packets on Serial port Arduino Nano

[Gennady]

Hi!

IDE 1.8.2, Nano, parsing lib https://github.com/bolderflight/UBLOX/tree/master/src

When parsing data coming from a GPS NEO M8N 200mS update, there are constant failures or data in general do not pass parsing at a packet size of more than 240 bytes (at speeds of 19200, 38400, 57600).I need 290 bytes! At other speeds I did not check, since they do not suit me. I tested with the revised HardwareSerial.h file with the RX buffer increased to 330 bytes.
Compiling a sketch gives: Global variables use 802 bytes (39%) of dynamic memory, leaving 1246 bytes for local variables. Maximum is 2048 bytes.

Code: [Select]


#if !defined(SERIAL_TX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64 //64
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 330 //16
#else
#define SERIAL_RX_BUFFER_SIZE 330 //64
#endif
#endif



The program in the cycle consists of two parts. The first part is the parsing of the 100 byte portion of the packet and the writing of values ​​into variables. After that, a trigger is set up that allows the operation of another part - my code.
My code does some transformations with the variables obtained above and produces the result on the UART TX. After that, the trigger is reset. Thus, my code is executed only after receiving new data through  library.

Code: [Select]



 
loop{

if (gps.readSensor())
    { digitalWrite(6, HIGH);
      Longitude_deg = gps.getLongitude_deg();
      Latitude_deg  = gps.getLatitude_deg();   
      GnssFixOk     = gps.isGnssFixOk();
      startLatitude_rad  = gps.getLatitude_rad();
      startLongitude_rad = gps.getLongitude_rad();
      MSLHeight_m   = gps.getMSLHeight_m();
      NumSatellites = gps.getNumSatellites();
      triggerParse = 1;
      digitalWrite(6, LOW);
    }

if (  triggerParse == 1 )
    { digitalWrite(9, HIGH);

//my code here....

      triggerParse = 0;
      digitalWrite(9, LOW);

    }
}



 Please advise what I can try to do?


I also give a picture for the case when everything works with a packet length of 240 bytes- channels from top to bottom: parsing work, UART RX, my code work, UART TX.


Thanks in advance!

Gennady

[Robin2]

Please post a complete program.

What is the purpose (or intended purpose) of the stuff in the first code block in your Original Post?

...R

[Deva_Rishi]

what type of variable do you use for parsing ? In fact yes post the complete sketch or at least the parsing functions...

[Gennady]

 I tested with the revised  file from //hardware/avr/arduino/ HardwareSerial.h with the RX buffer increased to 330 bytes:

Code: [Select]


#if !defined(SERIAL_TX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64 //64
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 330 //16
#else
#define SERIAL_RX_BUFFER_SIZE 330 //64
#endif
#endif



sketch

Code: [Select]


#include  "UBLOX.h"
#define DEBUG
#define UART_SPEED       57600

double   startLatitude_rad       = 0;
double   startLongitude_rad      = 0;
bool     GnssFixOk          = false;
double   Latitude_deg       = 0;
double   Longitude_deg      = 0;
double   Latitude_rad       = 0;
double   Longitude_rad      = 0;
double   MSLHeight_m        = 0;
uint8_t  NumSatellites      = 0;
double   GroundSpeed_ms     = 0;
double   pointLatitude_rad  = 0;
double   pointLongitude_rad = 0;

byte   triggerParse = 0;

UBLOX gps(Serial, UART_SPEED);


void setup()
{

  pinMode(6, OUTPUT);
  pinMode(7, OUTPUT);
  pinMode(8, OUTPUT);
  pinMode(9, OUTPUT);

  gps.begin();

#ifdef DEBUG
  Serial.println("START ");
  Serial.println("-------------------------- ");
#endif



}

void loop(){

  if (gps.readSensor())
  { digitalWrite(6, HIGH);
    Longitude_deg = gps.getLongitude_deg();
    Latitude_deg  = gps.getLatitude_deg();
    GnssFixOk     = gps.isGnssFixOk();
    startLatitude_rad  = gps.getLatitude_rad();
    startLongitude_rad = gps.getLongitude_rad();
    MSLHeight_m   = gps.getMSLHeight_m();
    NumSatellites = gps.getNumSatellites();
    triggerParse = 1;
    digitalWrite(6, LOW);
  }

  if (  triggerParse == 1 )
  { digitalWrite(9, HIGH);

#ifdef DEBUG
    Serial.print("N ");    Serial.print(NumSatellites);
    Serial.print(" F ");  Serial.print(GnssFixOk);
    Serial.print(" Lat ");  Serial.print(Latitude_deg, 7);
    Serial.print(" Lon ");  Serial.print(Longitude_deg, 7);
    Serial.print(" Spd "); Serial.print(GroundSpeed_ms);
#endif

    triggerParse = 0;
    digitalWrite(9, LOW);

  }
}


 Please advise what I can try to do?

[Gennady]

what type of variable do you use for parsing ? In fact yes post the complete sketch or at least the parsing functions...

in UART sending only bytes. Parsing also byte per byte- pasing function from .cpp file from lib:

Code: [Select]



/* reads packets from the uBlox receiver */
bool UBLOX::readSensor()
{
if (_parse(_ubxNavPvt_msgClass,_ubxNavPvt_msgId,_ubxNavPvt_msgLen)) {
return true;
} else {
return false;
}
}

/* parse the uBlox data */
bool UBLOX::_parse(uint8_t msg_class,uint8_t msg_id,uint16_t msg_length)
{
// read a byte from the serial port
while (_bus->available()) {
_byte = _bus->read();
// identify the packet header
if (_parserState < 2) {
if (_byte == _ubxHeader[_parserState]) {
_parserState++;
} else {
_parserState = 0;
}
} else {
if ((_parserState - 2) < msg_length) {
*((uint8_t *) &_tempPacket + _parserState - 2) = _byte;
}
_parserState++;
// compute checksum
if ((_parserState - 2) == msg_length) {
_calcChecksum(_checksum,((uint8_t *) &_tempPacket),msg_length);
} else if ((_parserState - 2) == (msg_length + 1)) {
if (_byte != _checksum[0]) {
_parserState = 0;
}
} else if ((_parserState - 2) == (msg_length + 2)) {
_parserState = 0;
if (_byte == _checksum[1]) {
memcpy(&_validPacket,&_tempPacket,sizeof(_validPacket));
return true;
}
} else if (_parserState > (msg_length + 4) ) {
_parserState = 0;
}
}
}
return false;
}

/* uBlox checksum */
void UBLOX::_calcChecksum(uint8_t* CK, uint8_t* payload, uint16_t length)
{
CK[0] = 0;
  CK[1] = 0;
for (uint8_t i = 0; i < length; i++) {
CK[0] += payload[i];
CK[1] += CK[0];
}
}



[Gennady]

Really nobody can advise?
Maybe this is a problem with the Stream class.
Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

[Robin2]

Or do I increase the buffer in 1.8.2, and the IDE takes the Hardwareserial.h file from 1.6.5, which is also installed by me?

As a piece of English that makes no sense.

And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

If you have a buffer that is bigger than the message that is being sent then I can think of no technical reason why the message cannot be parsed. However I know nothing about the UBLOX library you are using and that is where I would start searching for the problem.

Please post a link to the datasheet with the specification for the data that is SENT by the GPS device.

An example of a typical message would also be a big help. If it contains non-printing characters be sure to identify them all. They can be the key to effective parsing.

...R

[Gennady]

Thanks for the answer!
Oddly enough, the Serial library at compilation was taken from 163. I deleted 163. This did not help.

Today I experimented with the messages in the package. NEO 57600baud, 5Hz. The working version is the following and the only one.

UBX NAV-SOL, Size 60, 'Navigation Solution'
UBX NAV-PVT, Size 100, 'Navigation PVT Solution'
UBX NAV-POSLLH, Size 36, 'Geodetic Position'
UBX NAV-VELNED, Size 44, 'Velocity in WGS 84'

Total 240 bytes.

In this order, NEO builds them.

I add MON-HW2 long 36 byte. He stands in front of SOL on some time lead. But its frequency of appearance is 1 Hz. At this point, the parsing does not work. After that, I removed MON-HW2, but I inserted the message DOP 26 byte. It fell AFTER the POSLLH message. With him, the parsing also disappeared. At lower speeds 38400, 19200 the same.


I noticed that after pressing the reset button, the first parsing takes place under any conditions.
It seems that with a packet of more than 240 bytes, some kind of buffer overflows.

Maybe problem in Stream.h lib?

I do not know what to do anymore ((((



https://www.u-blox.com/sites/default/files/products/documents/u-blox8-M8_ReceiverDescrProtSpec_%28UBX-13003221%29_Public.pdf

[Robin2]

From a quick look at Section 32 of the linked PDF it seems that a message has the style
$GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E
and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

...R

[Gennady]

From a quick look at Section 32 of the linked PDF it seems that a message has the style
$GPGLL,4717.11634,N,00833.91297,E,124923.00,A,A*6E
and is terminated with a CR/LF

I would be surprised if any message is longer than 64 bytes - but you might like to point out which messages may be longer.

I reckon the second example in Serial Input Basics should receive those messages and the parse example can be adapted to do the parsing.

...R

No! It NMEA protocol. I use only UBX! Message some like this (up to 100byte and next Message start w/o any CR/LF :

//UBX-NAV-VELNED   
     
16:11:32  0000  B5 62 01 12 24 00 00 8B 39 1D 07 00 00 00 EC FF  µb$9ìÿ
              0010  FF FF DF FF FF FF 27 00 00 00 15 00 00 00 71 C2    ÿÿßÿÿÿ'qÂ
              0020  D1 01 83 00 00 00 3C 01 47 00 2C 60                    Ñ<G,`

//UBX-NAV-DOP 
16:11:33  0000  B5 62 01 02 1C 00 90 8C 39 1D 53 96 4D 12 96 DF  µb9SMß
              0010  05 1E 9B FC 01 00 0A 99 01 00 36 15 00 00 13 1D   ü6
              0020  00 00 28 3D                                                       (=

total packet up to 300byte and more.

[Deva_Rishi]

And if you have two versions of the IDE on your PC they will be completely separate. You can use whichever you choose. Recently when I needed a bigger serial input buffer I just made a copy of the IDE with a different name and changed the buffer size in that, Now I can use whichever version suits my requirement.

I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

[Robin2]

I thought i saw a way passing by on this forum that just puts an altered version of HardwareSerial.h in the project folder and the compiler will use that over the normal one.

I think I spent an hour with Google trying to figure that out and then the simple solution occurred to me - took all of 5 minutes.

...R

[Robin2]

No! It NMEA protocol. I use only UBX!

What section of the PDF is that in ?

Why do you seem to be using the more difficult system?

...R

[Gennady]

Thank you for your attention to my problem! Below are links to pages with descriptions of individual messages. The format UBX is good, compact, belongs to Ublox, I use it because it is used in autopilot.
NMEA it is very old format for ships. All the time I think that the problem is not in the program, but in the buffer or something else

33.17 UBX-NAV (0x01) ... 301

33.17.5 UBX-NAV-DOP (0x01 0x04) ........................................................................................ 304

33.17.13 UBX-NAV-POSLLH (0x01 0x02).................................................................................... 312
33.17.14 UBX-NAV-PVT (0x01 0x07) ........................................................................................ 313
33.17.20 UBX-NAV-SOL (0x01 0x06) ........................................................................................ 322
33.17.31 UBX-NAV-VELNED (0x01 0x12) .................................................................................. 337


I gave the parsing code above. As for me, everything is right there.
I read somewhere that Serial uses the Stream lib and the problem with receiving big data in the restrictions in Stream. "Some of the libraries that rely on Stream include : Serial..." Or the problem may be that the Serial lib buffer in the CPU RAM is full as a result of the program execution. And new data does not come. This can be confirmed by the fact that after pressing the reset button ALWAYS ONE parsing occurs. And then they are not, or there is very rarely, once a minute. This is with a packet of more than 240 bytes !!! Can a program in a loop have to clear buffer in the CPU RAM by a command Serialend () or another command? I do not know how well the Serial library is written for this case.

[Robin2]

The UBX frame structure is described on pages 134/5

With that type of message you need code to detect the start characters and then to interpret the length of the payload.


I don't know that I can give you any more assistance.

...R