Go Down

Topic: Proper method of receiving UART packet data far exceeding RX buffer size? (Read 265 times) previous topic - next topic

Rob_Mohr

I need to receive data from a device that blindly streams data at 115200 bps.  Each data packet is 3156 bytes in length.  The packet contains a unique 8 byte header and an 8 byte tail.

Standard methods of receiving data as shown in examples available through the IDE and those I have located on this forum, including the great tutorial by @Robin2, are working with incoming data packets of less than the default RX buffer size as far as I can understand.

I have read in various places that modifying HardwareSerial.h, or RingBuffer.h in the case of the Arduino Due, are not recommended and could lead to unexpected results.

Could someone please provide an example of how to deal with UART data exceeding the 64 or 128 byte defaults of the Arduino lineup?

Thanks,

Rob

UKHeliBob

Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

Rob_Mohr

Yes sir.  In fact I will be sending 10 such packets to a TCP/IP data logging server every 10 minutes.

pacoandres

The RX buffer is, as I understand, an temporal intermediate buffer. It isn't suposed to store the data until it's proccessed, it's designed to store the data just until the main task reads it.
You need an array to store the data while reading and work with it when finish.

Or you can proccess the data as it arrives.

Rob_Mohr

Yes, so how I have approached this is to let the RX buffer fill to 2X the packet size. 

Code: [Select]

while (1){
     if (Serial1.available() >= PACKET_SIZE*2 - 1){
      break;
     }
     else{
      continue;
     }
  }


I then transfer the buffer to an array. 


Code: [Select]

  for (int i = 0; i < PACKET_SIZE*2 - 1; i++){
    adc_data[i] = Serial1.read();
  }


I scan the array for the header.  Once I find the header I can use that index as my starting point for steaming the data to a modem on another UART.

Code: [Select]

  for (int j = 0;j < PACKET_SIZE*2 - 1;j++){
    if (adc_data[j] == 0x5A && adc_data[j+1] == 0xA5 && adc_data[j+2] == 0x55 && adc_data[j+3] == 0xAA){
      idx = j;  // Find header index
      break;
    }
  }


This all works, however to achieve it I must modify the default RX buffer size to 2X my packet size or 6312 bytes.

I'm searching for a more efficient way to approach this task without modifying the default IDE files for the target Arduino device.

UKHeliBob

As a starting point why not read the incoming data as it arrives until you find the header and only then save it to an array as each byte arrives ?
Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

Rob_Mohr

@UKHeliBob that's exactly the direction I am heading at the moment.  Given my novice C/C++ skills I've been trying to modify examples to achieve the result I want and this seems far enough outside the norm I haven't located any exemplars.

Once I figure out a clean generic way to approach this I will post for others to have as a reference.  Thank you for the advice.

Rob_Mohr

@UKHeliBob Okay, taking your suggestion I generated the following code:

Code: [Select]

#define PacketSize 3156

int MarkerLength = 4;
String Header = "5AA555AA";
String Tail = "EEFFEFFE";
boolean HeaderFound = false;
unsigned int adc_data[PacketSize];

void setup() {
  Serial.begin(9600);
  Serial1.begin(115200);
  Serial.println("Arduino ready to begin");
  delay(500);
}

void loop() {

  while (Serial1.available() >= MarkerLength) {
    int i = 0;
    adc_data[i] = Serial1.read();
    if (adc_data[i] == 0x5A) {
      i++;
      adc_data[i] = Serial1.read();
      if (adc_data[i] == 0xA5) {
        i++;
        adc_data[i] = Serial1.read();
        if (adc_data[i] == 0x55) {
          i++;
          adc_data[i] = Serial1.read();
          if (adc_data[i] == 0xAA) {
            HeaderFound = true;
            Serial.println("Found it!");
            for (int j = 0; j < MarkerLength; j++) {
              Serial.print(adc_data[j], HEX);
            }
            break;
          }
        }
      }
    }
  }
  if (HeaderFound == true) {
    int k = 4;
    while (k < PacketSize && Serial1.available() > 0) {
      adc_data[k] = Serial1.read();
      Serial.print(adc_data[k], HEX);
      k++;
    }
    Serial.println();
    // Go do something useful with adc_data[]
    memset(adc_data, 0, sizeof(adc_data));
    delay(500);
    HeaderFound = false;
  }
}


The output is:

Code: [Select]

18:02:26.438 -> Found it!
18:02:26.438 -> 5AA555AA
18:02:26.985 -> Found it!
18:02:27.019 -> 5AA555AA
18:02:27.564 -> Found it!
18:02:27.564 -> 5AA555AA
18:02:28.144 -> Found it!
18:02:28.144 -> 5AA555AA
18:02:28.689 -> Found it!
18:02:28.723 -> 5AA555AA


If I place "Serial.println("If HeaderFound evaluated to true");" following int k =4; and following the second while I put "Serial.println(k);" the output changes to:

Code: [Select]

18:27:39.641 -> Arduino ready to begin
18:27:40.392 -> Found it!
18:27:40.392 -> 5AA555AAIf HeaderFound evaluated to true
18:27:40.459 -> 4
18:27:40.459 -> 95
18:27:40.941 -> Found it!
18:27:40.976 -> 5AA555AAIf HeaderFound evaluated to true
18:27:41.010 -> 4
18:27:41.010 -> 9C
18:27:41.524 -> Found it!
18:27:41.524 -> 5AA555AAIf HeaderFound evaluated to true
18:27:41.593 -> 4
18:27:41.593 -> A3


Which would indicate that the while loop is exiting after one pass through.  Is Serial1(available) re-evaluated on each pass through the while?

Any thoughts on how to fix this?

PS:  For these tests I have returned RingBuffer.h to the original file, i.e. #define SERIAL_BUFFER_SIZE 128,

Rob

Blackfin

Rob, how about something like this:

Code: [Select]

#define MSG_HDR_LEN     4
#define MSG_TRLR_LEN    4
#define MSG_PCKT_SIZE   3156        //does 3156 include header and trailer size? If so, reduce by (MSG_HDR_LEN+MSG_TRLR_LEN)

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

byte
    grPayload[MSG_PCKT_SIZE];
   
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");   
   
}//setup

void loop( void )
{
    RxStateMachine();
   
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static int
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
           
            switch( stateRx )
            {
                case    ST_HDR:
                    if( cByte == cgrHdr[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_HDR_LEN-1 )
                        {
                            Serial.println("DBG: Got header.");       
                            //                       
                            stateRx = ST_PAYLOAD;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        nIdx = 0;
                       
                    }//else
           
                break;

                case    ST_PAYLOAD:
                    grPayload[nIdx++] = cByte;
                    if( nIdx == MSG_PCKT_SIZE-1 )
                    {
                        Serial.println("DBG: Got payload.");       
                        //
                        stateRx = ST_TRAILER;
                        nIdx = 0;
                       
                    }//if
                                       
                break;

                case    ST_TRAILER:
                    //almost a copy of looking for the header except in the event
                    //of failure/mismatch, immediately return to look for a header
                    if( cByte == cgrTrailer[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_TRLR_LEN-1 )
                        {
                            Serial.println("DBG: Got message trailer.");       
                            //
                            stateRx = ST_HDR;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        //if we get an error receiving the trailer, immediately go back to look
                        //for another header.   
                        Serial.println("DBG: Got bad trailer.");
                        nIdx = 0;
                        stateRx = ST_HDR;
                       
                    }//else
               
                break;
       
            }//switch
           
        }//while
       
    }//if
   
}//RxStateMachine


Note you may need to make adjustments (see comment re PAYLOAD size.)

A general idea here is to not wait for the buffer to be above a certain size; just grind through bites as quickly as they come in and let the state machine naturally flow.

Rob_Mohr

@Blackfin,

This looks really promising.  This is a much more elegant way of going about this.  The data format is:

Header  - 4 bytes of char, constant
Data1   - unsigned short(2 byte) x 500
Data2   - unsigned short(2 byte) x 176
Data3   - unsigned short(2 byte) x 400
Data4  -  unsigned short(2 byte) x 276
Data5  -  unsigned short(2 byte) x 208
Data6  -  float(4 byte) x 1
Data7  -  float(4 byte) x 1
Data8  -  float(4 byte) x 1
Data9  -  float(4 byte) x 1
Data10 - float(4 byte) x 1
Data11 - float(4 byte) x 1
Data12 - float(4 byte) x 1
Tail       - 4 bytes of char, constant

So, I have a total of 3156 bytes and 3148 bytes for the payload.  Making the appropriate change to MSG_PCKT_SIZE = 3148 and running the code I get the following output:


Code: [Select]

20:15:32.508 -> Arduino ready to begin
20:15:32.714 -> DBG: Got header.
20:15:32.986 -> DBG: Got payload.
20:15:33.020 -> DBG: Got bad trailer.
20:15:33.055 -> DBG: Got header.
20:15:33.290 -> DBG: Got payload.
20:15:33.290 -> DBG: Got bad trailer.


I didn't think it would make any difference what the format of the payload bytes were, if we are just packing them away as bytes, however it appears that we aren't aligned to gather a valid tail.

Would it add too much delay to Serial.print(data,HEX) in each state in order to debug what I'm seeing in incoming data?

My previous code which runs with SERIAL_BUFFER_SIZE 6312 in RingBuffer.h (C:\Users\<USERNAME>\AppData\Local\Arduino15\packages\arduino\hardware\sam\1.6.12\cores\arduino) for Arduino Due is as follows:

Code: [Select]

#define PACKET_SIZE 3156
// ADC_PACKET_HEAD   0x5AA555AA
// ADC_PACKET_TAIL   0xEEFFEFFE

int idx;
unsigned int adc_data[PACKET_SIZE];

void setup() {
  Serial.begin(9600);
  Serial1.begin(115200);
  Serial.println("Arduino Ready to recieve");
}

void loop() {
  while (1){
     if (Serial1.available() >= PACKET_SIZE*2 - 1){
      break;
     }
     else{
      continue;
     }
  }
  for (int i = 0; i < PACKET_SIZE*2 - 1; i++){
    adc_data[i] = Serial1.read();
  }

  for (int j = 0;j < PACKET_SIZE*2 - 1;j++){
    if (adc_data[j] == 0x5A && adc_data[j+1] == 0xA5 && adc_data[j+2] == 0x55 && adc_data[j+3] == 0xAA){
      idx = j;  // Find header index
      break;
    }
  }
  Serial.println(adc_data[idx+PACKET_SIZE-1],HEX); // If this printout number is 0xFE, that means we received one data packet successfully.
}
}


Which generates this output:

Code: [Select]

US-D1 Readout
22:14:46.108 -> FE
22:14:46.688 -> FE
22:14:47.242 -> FE
22:14:47.830 -> FE
22:14:48.382 -> FE
22:14:48.966 -> FE
22:14:49.513 -> FE
22:14:50.100 -> FE
22:14:50.655 -> FE
22:14:51.241 -> FE


While this *does* work, albeit ugly, it will only work with the modified RX buffer size which appears to cause problems with the Wire library which I will need for the RTC communications.  If I can get your code properly aligned to the packet size It looks like I'll be golden.

Robin2

I need to receive data from a device that blindly streams data at 115200 bps.  Each data packet is 3156 bytes in length.  The packet contains a unique 8 byte header and an 8 byte tail.
Yes sir.  In fact I will be sending 10 such packets to a TCP/IP data logging server every 10 minutes.
What exactly is the project?

What Arduino are you using?

Maybe a RaspberryPi would be more suitable?

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Juraj

has the device a flow control pin?

Blackfin

I've changed the indexing a bit and added a debug print at the end. How's this:

Code: [Select]

#define MSG_HDR_LEN         4
#define MSG_TRLR_LEN        4
#define MSG_PAYLOAD_SIZE    3148
//to make room for 3148 byte payload plus HDR and TRLR
#define MSG_PCKT_SIZE       (MSG_HDR_LEN+MSG_PAYLOAD_SIZE+MSG_TRLR_LEN) 

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

byte
    grHeader[MSG_HDR_LEN];
byte
    grTrailer[MSG_TRLR_LEN];
byte
    grPayload[MSG_PAYLOAD_SIZE];
   
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");   

    //clear the payload buffer
    memset( grHeader, 0, sizeof( grHeader ) );
    memset( grPayload, 0, sizeof( grPayload) );
    memset( grTrailer, 0, sizeof( grTrailer ) );
   
}//setup

void loop( void )
{
    RxStateMachine();
   
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static int
        nHdrIdx = 0,
        nTrlrIdx = 0,
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
           
            switch( stateRx )
            {
                //0x5A, 0xA5, 0x55, 0xAA
                //     1     2     3     4
                case    ST_HDR:       
                    grHeader[nIdx] = cByte;
                    if( cByte == cgrHdr[nIdx] )
                    {                       
                        nIdx++;
                        if( nIdx == MSG_HDR_LEN )
                        {
                            Serial.println("DBG: Got header.");       
                            //                       
                            stateRx = ST_PAYLOAD;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        nIdx = 0;
                       
                    }//else
           
                break;

                case    ST_PAYLOAD:
                    grPayload[nIdx++] = cByte;
                    if( nIdx == MSG_PAYLOAD_SIZE )
                    {
                        Serial.println("DBG: Got payload.");       
                        //
                        stateRx = ST_TRAILER;
                        nIdx = 0;
                       
                    }//if
                                       
                break;

                case    ST_TRAILER:
                    //almost a copy of looking for the header except in the event
                    //of failure/mismatch, immediately return to look for a header
                    grTrailer[nIdx] = cByte;
                    if( cByte == cgrTrailer[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_TRLR_LEN )
                        {
                            Serial.println("DBG: Got message trailer.");       
                            //
                            stateRx = ST_HDR;
                            nIdx = 0;
                            DebugPrintPacket();
                                                     
                        }//if
                       
                    }//if
                    else
                    {
                        //if we get an error receiving the trailer, immediately go back to look
                        //for another header.   
                        Serial.println("DBG: Got bad trailer.");
                        nIdx = 0;
                        stateRx = ST_HDR;
                        DebugPrintPacket();
                       
                    }//else
               
                break;
       
            }//switch
           
        }//while
       
    }//if
   
}//RxStateMachine

void DebugPrintPacket( void )
{
    //header
    Serial.print( "HDR... " );
    for( int i=0; i<MSG_HDR_LEN; i++ )
    {
        Serial.print( grHeader[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "\n" );
   
    //payload
    //due to size, only print the first 25 bytes...
    Serial.print( "PYLD.. " );
    for( int i=0; i<25; i++ )
    {
        Serial.print( grPayload[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "...\n" );

    //trailer
    Serial.print( "TRLR.. " );
    for( int i=0; i<MSG_TRLR_LEN; i++ )
    {
        Serial.print( grTrailer[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "\n" );

    //clear buffers for next msg
    memset( grHeader, 0, sizeof( grHeader ) );
    memset( grPayload, 0, sizeof( grPayload) );
    memset( grTrailer, 0, sizeof( grTrailer ) );

}//DebugPrintPacket

Blackfin

And just to try (compiles, not tested) a variant that is supposed to parse and print the packet.

Note that the debug print process will take a while; you may have to double-buffer and re-work the print if you want to continue to receive data; for debug it's just intended to show you're rxing data...)

Code: [Select]
#define MSG_HDR_LEN         4
#define MSG_TRLR_LEN        4
#define MSG_PAYLOAD_SIZE    3148

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

typedef struct structPacket
{
    byte            grbHeader[MSG_HDR_LEN];
    unsigned int    grnData1[500];
    unsigned int    grnData2[176];
    unsigned int    grnData3[400];
    unsigned int    grnData4[276];
    unsigned int    grnData5[208];
    float           fData6;
    float           fData7;
    float           fData8;
    float           fData9;
    float           fData10;
    float           fData11;
    float           fData12;
    byte            grbTrailer[MSG_TRLR_LEN];
    
}s_Packet;

typedef union unionPacket
{
    s_Packet    sPacket;
    byte        grPacket[MSG_HDR_LEN+MSG_PAYLOAD_SIZE+MSG_TRLR_LEN];
    
}u_Packet;

u_Packet Packet;
    
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");    

    Packet.grPacket[0] = 0x5a;
    Packet.grPacket[1] = 0xa5;
    Packet.grPacket[2] = 0xaa;
    Packet.grPacket[3] = 0x55;

    Serial.print( Packet.sPacket.grbHeader[0], HEX );
    Serial.print( Packet.sPacket.grbHeader[1], HEX );
    Serial.print( Packet.sPacket.grbHeader[2], HEX );
    Serial.print( Packet.sPacket.grbHeader[3], HEX );
    Serial.println();

}//setup

void loop( void )
{
    RxStateMachine();
    
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static bool
        bTrailerGood;
    static int
        nSegIdx = 0,
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
            Packet.grPacket[nIdx++] = cByte;
            
            switch( stateRx )
            {
                //0x5A, 0xA5, 0x55, 0xAA
                //     1     2     3     4
                case    ST_HDR:                
                    if( cByte == cgrHdr[nSegIdx] )
                    {                        
                        nSegIdx++;
                        if( nSegIdx == MSG_HDR_LEN )
                        {
                            Serial.println("DBG: Got header.");        
                            //                        
                            stateRx = ST_PAYLOAD;
                            nSegIdx = 0;
                            
                        }//if
                        
                    }//if
                    else
                    {
                        nIdx = 0;
                        nSegIdx = 0;
                        
                    }//else
            
                break;

                case    ST_PAYLOAD:
                    nSegIdx++;                  
                    if( nSegIdx == MSG_PAYLOAD_SIZE )
                    {
                        Serial.println("DBG: Got payload.");        
                        //
                        stateRx = ST_TRAILER;
                        bTrailerGood = true;
                        nSegIdx = 0;
                        
                    }//if
                                        
                break;

                case    ST_TRAILER:                  
                    if( cByte == cgrTrailer[nSegIdx] )
                    {
                        if( nSegIdx == MSG_TRLR_LEN )
                        {
                            if( bTrailerGood )
                                Serial.println("DBG: Got message trailer.");        
                            else
                                Serial.println("DBG: Got bad message trailer.");        
                            //
                            stateRx = ST_HDR;
                            nSegIdx = 0;
                            nIdx = 0;
                            //
                            DebugPrintPacket();
                                                    
                        }//if
                        
                    }//if
                    else
                    {
                        bTrailerGood = false;
                        
                    }//else

                    nSegIdx++;
                                    
                break;
        
            }//switch
            
        }//while
        
    }//if
    
}//RxStateMachine

void DebugPrintPacket( void )
{
    //header
    Serial.print( "HDR... " );
    Serial.print( Packet.sPacket.grbHeader[0], HEX );
    Serial.print( Packet.sPacket.grbHeader[1], HEX );
    Serial.print( Packet.sPacket.grbHeader[2], HEX );
    Serial.println( Packet.sPacket.grbHeader[3], HEX );

    //payload
    Serial.print( "DATA1:\n" );
    for( int i=0; i<500; i++ )
        Serial.println( Packet.sPacket.grnData1[i], DEC );
    Serial.print( "DATA2:\n" );
    for( int i=0; i<176; i++ )
        Serial.println( Packet.sPacket.grnData2[i], DEC );
    Serial.print( "DATA3:\n" );
    for( int i=0; i<400; i++ )
        Serial.println( Packet.sPacket.grnData3[i], DEC );
    Serial.print( "DATA4:\n" );
    for( int i=0; i<276; i++ )
        Serial.println( Packet.sPacket.grnData4[i], DEC );
    Serial.print( "DATA5:\n" );
    for( int i=0; i<208; i++ )
        Serial.println( Packet.sPacket.grnData5[i], DEC );

    Serial.print( "DATA6 : " );
    Serial.println( Packet.sPacket.fData6, 3 );
    Serial.print( "DATA7 : " );
    Serial.println( Packet.sPacket.fData7, 3 );
    Serial.print( "DATA8 : " );
    Serial.println( Packet.sPacket.fData8, 3 );
    Serial.print( "DATA9 : " );
    Serial.println( Packet.sPacket.fData9, 3 );
    Serial.print( "DATA10: " );
    Serial.println( Packet.sPacket.fData10, 3 );
    Serial.print( "DATA11:" );
    Serial.println( Packet.sPacket.fData11, 3 );
    Serial.print( "DATA12:" );
    Serial.println( Packet.sPacket.fData12, 3 );
    
    Serial.print( "TRLR.. " );
    Serial.print( Packet.sPacket.grbTrailer[0], HEX );
    Serial.print( Packet.sPacket.grbTrailer[1], HEX );
    Serial.print( Packet.sPacket.grbTrailer[2], HEX );
    Serial.println( Packet.sPacket.grbTrailer[3], HEX );

}//DebugPrintPacket

Rob_Mohr

What exactly is the project?

What Arduino are you using?

Maybe a RaspberryPi would be more suitable?

...R
@Robin2,

I have a prototype sensor whose firmware has been set up to generate a bunch of raw debug output.  I need to setup a data logger to capture this output with the sensor placed in real-world conditions and then send the data over a cellular connection to be logged.  The data will be used to fine tune algorithms associated with the sensor.

I am using the Arduino Due.

A RaspberryPi might have been more suitable in terms of dealing with the amount of data to be transferred, however I'll be dealing with a very tight power budget.  I also don't really need all the overhead that the Pi brings with it.

Rob

Go Up