Arduino Forum

Using Arduino => Programming Questions => Topic started by: Rob_Mohr on Sep 22, 2019, 10:03 pm

Title: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 22, 2019, 10:03 pm
I need to receive data from a device that blindly streams data at 115200 bps.  Each data packet is 3156 bytes in length.  The packet contains a unique 8 byte header and an 8 byte tail.

Standard methods of receiving data as shown in examples available through the IDE and those I have located on this forum, including the great tutorial by @Robin2, are working with incoming data packets of less than the default RX buffer size as far as I can understand.

I have read in various places that modifying HardwareSerial.h, or RingBuffer.h in the case of the Arduino Due, are not recommended and could lead to unexpected results.

Could someone please provide an example of how to deal with UART data exceeding the 64 or 128 byte defaults of the Arduino lineup?

Thanks,

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: UKHeliBob on Sep 22, 2019, 10:07 pm
Do you actually need all of the 3156  bytes ?
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 22, 2019, 10:09 pm
Yes sir.  In fact I will be sending 10 such packets to a TCP/IP data logging server every 10 minutes.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: pacoandres on Sep 22, 2019, 10:44 pm
The RX buffer is, as I understand, an temporal intermediate buffer. It isn't suposed to store the data until it's proccessed, it's designed to store the data just until the main task reads it.
You need an array to store the data while reading and work with it when finish.

Or you can proccess the data as it arrives.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 22, 2019, 10:57 pm
Yes, so how I have approached this is to let the RX buffer fill to 2X the packet size. 

Code: [Select]

while (1){
     if (Serial1.available() >= PACKET_SIZE*2 - 1){
      break;
     }
     else{
      continue;
     }
  }


I then transfer the buffer to an array. 


Code: [Select]

  for (int i = 0; i < PACKET_SIZE*2 - 1; i++){
    adc_data[i] = Serial1.read();
  }


I scan the array for the header.  Once I find the header I can use that index as my starting point for steaming the data to a modem on another UART.

Code: [Select]

  for (int j = 0;j < PACKET_SIZE*2 - 1;j++){
    if (adc_data[j] == 0x5A && adc_data[j+1] == 0xA5 && adc_data[j+2] == 0x55 && adc_data[j+3] == 0xAA){
      idx = j;  // Find header index
      break;
    }
  }


This all works, however to achieve it I must modify the default RX buffer size to 2X my packet size or 6312 bytes.

I'm searching for a more efficient way to approach this task without modifying the default IDE files for the target Arduino device.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: UKHeliBob on Sep 22, 2019, 11:05 pm
As a starting point why not read the incoming data as it arrives until you find the header and only then save it to an array as each byte arrives ?
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 22, 2019, 11:12 pm
@UKHeliBob that's exactly the direction I am heading at the moment.  Given my novice C/C++ skills I've been trying to modify examples to achieve the result I want and this seems far enough outside the norm I haven't located any exemplars.

Once I figure out a clean generic way to approach this I will post for others to have as a reference.  Thank you for the advice.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 23, 2019, 03:41 am
@UKHeliBob Okay, taking your suggestion I generated the following code:

Code: [Select]

#define PacketSize 3156

int MarkerLength = 4;
String Header = "5AA555AA";
String Tail = "EEFFEFFE";
boolean HeaderFound = false;
unsigned int adc_data[PacketSize];

void setup() {
  Serial.begin(9600);
  Serial1.begin(115200);
  Serial.println("Arduino ready to begin");
  delay(500);
}

void loop() {

  while (Serial1.available() >= MarkerLength) {
    int i = 0;
    adc_data[i] = Serial1.read();
    if (adc_data[i] == 0x5A) {
      i++;
      adc_data[i] = Serial1.read();
      if (adc_data[i] == 0xA5) {
        i++;
        adc_data[i] = Serial1.read();
        if (adc_data[i] == 0x55) {
          i++;
          adc_data[i] = Serial1.read();
          if (adc_data[i] == 0xAA) {
            HeaderFound = true;
            Serial.println("Found it!");
            for (int j = 0; j < MarkerLength; j++) {
              Serial.print(adc_data[j], HEX);
            }
            break;
          }
        }
      }
    }
  }
  if (HeaderFound == true) {
    int k = 4;
    while (k < PacketSize && Serial1.available() > 0) {
      adc_data[k] = Serial1.read();
      Serial.print(adc_data[k], HEX);
      k++;
    }
    Serial.println();
    // Go do something useful with adc_data[]
    memset(adc_data, 0, sizeof(adc_data));
    delay(500);
    HeaderFound = false;
  }
}


The output is:

Code: [Select]

18:02:26.438 -> Found it!
18:02:26.438 -> 5AA555AA
18:02:26.985 -> Found it!
18:02:27.019 -> 5AA555AA
18:02:27.564 -> Found it!
18:02:27.564 -> 5AA555AA
18:02:28.144 -> Found it!
18:02:28.144 -> 5AA555AA
18:02:28.689 -> Found it!
18:02:28.723 -> 5AA555AA


If I place "Serial.println("If HeaderFound evaluated to true");" following int k =4; and following the second while I put "Serial.println(k);" the output changes to:

Code: [Select]

18:27:39.641 -> Arduino ready to begin
18:27:40.392 -> Found it!
18:27:40.392 -> 5AA555AAIf HeaderFound evaluated to true
18:27:40.459 -> 4
18:27:40.459 -> 95
18:27:40.941 -> Found it!
18:27:40.976 -> 5AA555AAIf HeaderFound evaluated to true
18:27:41.010 -> 4
18:27:41.010 -> 9C
18:27:41.524 -> Found it!
18:27:41.524 -> 5AA555AAIf HeaderFound evaluated to true
18:27:41.593 -> 4
18:27:41.593 -> A3


Which would indicate that the while loop is exiting after one pass through.  Is Serial1(available) re-evaluated on each pass through the while?

Any thoughts on how to fix this?

PS:  For these tests I have returned RingBuffer.h to the original file, i.e. #define SERIAL_BUFFER_SIZE 128,

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Blackfin on Sep 23, 2019, 04:34 am
Rob, how about something like this:

Code: [Select]

#define MSG_HDR_LEN     4
#define MSG_TRLR_LEN    4
#define MSG_PCKT_SIZE   3156        //does 3156 include header and trailer size? If so, reduce by (MSG_HDR_LEN+MSG_TRLR_LEN)

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

byte
    grPayload[MSG_PCKT_SIZE];
   
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");   
   
}//setup

void loop( void )
{
    RxStateMachine();
   
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static int
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
           
            switch( stateRx )
            {
                case    ST_HDR:
                    if( cByte == cgrHdr[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_HDR_LEN-1 )
                        {
                            Serial.println("DBG: Got header.");       
                            //                       
                            stateRx = ST_PAYLOAD;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        nIdx = 0;
                       
                    }//else
           
                break;

                case    ST_PAYLOAD:
                    grPayload[nIdx++] = cByte;
                    if( nIdx == MSG_PCKT_SIZE-1 )
                    {
                        Serial.println("DBG: Got payload.");       
                        //
                        stateRx = ST_TRAILER;
                        nIdx = 0;
                       
                    }//if
                                       
                break;

                case    ST_TRAILER:
                    //almost a copy of looking for the header except in the event
                    //of failure/mismatch, immediately return to look for a header
                    if( cByte == cgrTrailer[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_TRLR_LEN-1 )
                        {
                            Serial.println("DBG: Got message trailer.");       
                            //
                            stateRx = ST_HDR;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        //if we get an error receiving the trailer, immediately go back to look
                        //for another header.   
                        Serial.println("DBG: Got bad trailer.");
                        nIdx = 0;
                        stateRx = ST_HDR;
                       
                    }//else
               
                break;
       
            }//switch
           
        }//while
       
    }//if
   
}//RxStateMachine


Note you may need to make adjustments (see comment re PAYLOAD size.)

A general idea here is to not wait for the buffer to be above a certain size; just grind through bites as quickly as they come in and let the state machine naturally flow.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 23, 2019, 07:25 am
@Blackfin,

This looks really promising.  This is a much more elegant way of going about this.  The data format is:

Header  - 4 bytes of char, constant
Data1   - unsigned short(2 byte) x 500
Data2   - unsigned short(2 byte) x 176
Data3   - unsigned short(2 byte) x 400
Data4  -  unsigned short(2 byte) x 276
Data5  -  unsigned short(2 byte) x 208
Data6  -  float(4 byte) x 1
Data7  -  float(4 byte) x 1
Data8  -  float(4 byte) x 1
Data9  -  float(4 byte) x 1
Data10 - float(4 byte) x 1
Data11 - float(4 byte) x 1
Data12 - float(4 byte) x 1
Tail       - 4 bytes of char, constant

So, I have a total of 3156 bytes and 3148 bytes for the payload.  Making the appropriate change to MSG_PCKT_SIZE = 3148 and running the code I get the following output:


Code: [Select]

20:15:32.508 -> Arduino ready to begin
20:15:32.714 -> DBG: Got header.
20:15:32.986 -> DBG: Got payload.
20:15:33.020 -> DBG: Got bad trailer.
20:15:33.055 -> DBG: Got header.
20:15:33.290 -> DBG: Got payload.
20:15:33.290 -> DBG: Got bad trailer.


I didn't think it would make any difference what the format of the payload bytes were, if we are just packing them away as bytes, however it appears that we aren't aligned to gather a valid tail.

Would it add too much delay to Serial.print(data,HEX) in each state in order to debug what I'm seeing in incoming data?

My previous code which runs with SERIAL_BUFFER_SIZE 6312 in RingBuffer.h (C:\Users\<USERNAME>\AppData\Local\Arduino15\packages\arduino\hardware\sam\1.6.12\cores\arduino) for Arduino Due is as follows:

Code: [Select]

#define PACKET_SIZE 3156
// ADC_PACKET_HEAD   0x5AA555AA
// ADC_PACKET_TAIL   0xEEFFEFFE

int idx;
unsigned int adc_data[PACKET_SIZE];

void setup() {
  Serial.begin(9600);
  Serial1.begin(115200);
  Serial.println("Arduino Ready to recieve");
}

void loop() {
  while (1){
     if (Serial1.available() >= PACKET_SIZE*2 - 1){
      break;
     }
     else{
      continue;
     }
  }
  for (int i = 0; i < PACKET_SIZE*2 - 1; i++){
    adc_data[i] = Serial1.read();
  }

  for (int j = 0;j < PACKET_SIZE*2 - 1;j++){
    if (adc_data[j] == 0x5A && adc_data[j+1] == 0xA5 && adc_data[j+2] == 0x55 && adc_data[j+3] == 0xAA){
      idx = j;  // Find header index
      break;
    }
  }
  Serial.println(adc_data[idx+PACKET_SIZE-1],HEX); // If this printout number is 0xFE, that means we received one data packet successfully.
}
}


Which generates this output:

Code: [Select]

US-D1 Readout
22:14:46.108 -> FE
22:14:46.688 -> FE
22:14:47.242 -> FE
22:14:47.830 -> FE
22:14:48.382 -> FE
22:14:48.966 -> FE
22:14:49.513 -> FE
22:14:50.100 -> FE
22:14:50.655 -> FE
22:14:51.241 -> FE


While this *does* work, albeit ugly, it will only work with the modified RX buffer size which appears to cause problems with the Wire library which I will need for the RTC communications.  If I can get your code properly aligned to the packet size It looks like I'll be golden.
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Robin2 on Sep 23, 2019, 09:06 am
I need to receive data from a device that blindly streams data at 115200 bps.  Each data packet is 3156 bytes in length.  The packet contains a unique 8 byte header and an 8 byte tail.
Yes sir.  In fact I will be sending 10 such packets to a TCP/IP data logging server every 10 minutes.
What exactly is the project?

What Arduino are you using?

Maybe a RaspberryPi would be more suitable?

...R
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Juraj on Sep 23, 2019, 11:53 am
has the device a flow control pin?
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Blackfin on Sep 23, 2019, 01:52 pm
I've changed the indexing a bit and added a debug print at the end. How's this:

Code: [Select]

#define MSG_HDR_LEN         4
#define MSG_TRLR_LEN        4
#define MSG_PAYLOAD_SIZE    3148
//to make room for 3148 byte payload plus HDR and TRLR
#define MSG_PCKT_SIZE       (MSG_HDR_LEN+MSG_PAYLOAD_SIZE+MSG_TRLR_LEN) 

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

byte
    grHeader[MSG_HDR_LEN];
byte
    grTrailer[MSG_TRLR_LEN];
byte
    grPayload[MSG_PAYLOAD_SIZE];
   
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");   

    //clear the payload buffer
    memset( grHeader, 0, sizeof( grHeader ) );
    memset( grPayload, 0, sizeof( grPayload) );
    memset( grTrailer, 0, sizeof( grTrailer ) );
   
}//setup

void loop( void )
{
    RxStateMachine();
   
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static int
        nHdrIdx = 0,
        nTrlrIdx = 0,
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
           
            switch( stateRx )
            {
                //0x5A, 0xA5, 0x55, 0xAA
                //     1     2     3     4
                case    ST_HDR:       
                    grHeader[nIdx] = cByte;
                    if( cByte == cgrHdr[nIdx] )
                    {                       
                        nIdx++;
                        if( nIdx == MSG_HDR_LEN )
                        {
                            Serial.println("DBG: Got header.");       
                            //                       
                            stateRx = ST_PAYLOAD;
                            nIdx = 0;
                           
                        }//if
                       
                    }//if
                    else
                    {
                        nIdx = 0;
                       
                    }//else
           
                break;

                case    ST_PAYLOAD:
                    grPayload[nIdx++] = cByte;
                    if( nIdx == MSG_PAYLOAD_SIZE )
                    {
                        Serial.println("DBG: Got payload.");       
                        //
                        stateRx = ST_TRAILER;
                        nIdx = 0;
                       
                    }//if
                                       
                break;

                case    ST_TRAILER:
                    //almost a copy of looking for the header except in the event
                    //of failure/mismatch, immediately return to look for a header
                    grTrailer[nIdx] = cByte;
                    if( cByte == cgrTrailer[nIdx] )
                    {
                        nIdx++;
                        if( nIdx == MSG_TRLR_LEN )
                        {
                            Serial.println("DBG: Got message trailer.");       
                            //
                            stateRx = ST_HDR;
                            nIdx = 0;
                            DebugPrintPacket();
                                                     
                        }//if
                       
                    }//if
                    else
                    {
                        //if we get an error receiving the trailer, immediately go back to look
                        //for another header.   
                        Serial.println("DBG: Got bad trailer.");
                        nIdx = 0;
                        stateRx = ST_HDR;
                        DebugPrintPacket();
                       
                    }//else
               
                break;
       
            }//switch
           
        }//while
       
    }//if
   
}//RxStateMachine

void DebugPrintPacket( void )
{
    //header
    Serial.print( "HDR... " );
    for( int i=0; i<MSG_HDR_LEN; i++ )
    {
        Serial.print( grHeader[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "\n" );
   
    //payload
    //due to size, only print the first 25 bytes...
    Serial.print( "PYLD.. " );
    for( int i=0; i<25; i++ )
    {
        Serial.print( grPayload[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "...\n" );

    //trailer
    Serial.print( "TRLR.. " );
    for( int i=0; i<MSG_TRLR_LEN; i++ )
    {
        Serial.print( grTrailer[i], HEX );
        Serial.print( " " );
    }
    Serial.print( "\n" );

    //clear buffers for next msg
    memset( grHeader, 0, sizeof( grHeader ) );
    memset( grPayload, 0, sizeof( grPayload) );
    memset( grTrailer, 0, sizeof( grTrailer ) );

}//DebugPrintPacket
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Blackfin on Sep 23, 2019, 03:31 pm
And just to try (compiles, not tested) a variant that is supposed to parse and print the packet.

Note that the debug print process will take a while; you may have to double-buffer and re-work the print if you want to continue to receive data; for debug it's just intended to show you're rxing data...)

Code: [Select]
#define MSG_HDR_LEN         4
#define MSG_TRLR_LEN        4
#define MSG_PAYLOAD_SIZE    3148

const byte cgrHdr[MSG_HDR_LEN] = { 0x5A, 0xA5, 0x55, 0xAA };
const byte cgrTrailer[MSG_TRLR_LEN] = { 0xEE, 0xFF, 0xEF, 0xFE };

typedef struct structPacket
{
    byte            grbHeader[MSG_HDR_LEN];
    unsigned int    grnData1[500];
    unsigned int    grnData2[176];
    unsigned int    grnData3[400];
    unsigned int    grnData4[276];
    unsigned int    grnData5[208];
    float           fData6;
    float           fData7;
    float           fData8;
    float           fData9;
    float           fData10;
    float           fData11;
    float           fData12;
    byte            grbTrailer[MSG_TRLR_LEN];
    
}s_Packet;

typedef union unionPacket
{
    s_Packet    sPacket;
    byte        grPacket[MSG_HDR_LEN+MSG_PAYLOAD_SIZE+MSG_TRLR_LEN];
    
}u_Packet;

u_Packet Packet;
    
void setup( void )
{
    Serial.begin(9600);
    Serial1.begin(115200);
    Serial.println("Arduino ready to begin");    

    Packet.grPacket[0] = 0x5a;
    Packet.grPacket[1] = 0xa5;
    Packet.grPacket[2] = 0xaa;
    Packet.grPacket[3] = 0x55;

    Serial.print( Packet.sPacket.grbHeader[0], HEX );
    Serial.print( Packet.sPacket.grbHeader[1], HEX );
    Serial.print( Packet.sPacket.grbHeader[2], HEX );
    Serial.print( Packet.sPacket.grbHeader[3], HEX );
    Serial.println();

}//setup

void loop( void )
{
    RxStateMachine();
    
}//loop

//state names
#define ST_HDR      0
#define ST_PAYLOAD  1
#define ST_TRAILER  2
//
void RxStateMachine( void )
{
    byte
        cByte;
    static bool
        bTrailerGood;
    static int
        nSegIdx = 0,
        nIdx = 0;
    static byte
        stateRx = ST_HDR;

    if( Serial.available() > 0 )
    {
        while( Serial.available() )
        {
            cByte = Serial.read();
            Packet.grPacket[nIdx++] = cByte;
            
            switch( stateRx )
            {
                //0x5A, 0xA5, 0x55, 0xAA
                //     1     2     3     4
                case    ST_HDR:                
                    if( cByte == cgrHdr[nSegIdx] )
                    {                        
                        nSegIdx++;
                        if( nSegIdx == MSG_HDR_LEN )
                        {
                            Serial.println("DBG: Got header.");        
                            //                        
                            stateRx = ST_PAYLOAD;
                            nSegIdx = 0;
                            
                        }//if
                        
                    }//if
                    else
                    {
                        nIdx = 0;
                        nSegIdx = 0;
                        
                    }//else
            
                break;

                case    ST_PAYLOAD:
                    nSegIdx++;                  
                    if( nSegIdx == MSG_PAYLOAD_SIZE )
                    {
                        Serial.println("DBG: Got payload.");        
                        //
                        stateRx = ST_TRAILER;
                        bTrailerGood = true;
                        nSegIdx = 0;
                        
                    }//if
                                        
                break;

                case    ST_TRAILER:                  
                    if( cByte == cgrTrailer[nSegIdx] )
                    {
                        if( nSegIdx == MSG_TRLR_LEN )
                        {
                            if( bTrailerGood )
                                Serial.println("DBG: Got message trailer.");        
                            else
                                Serial.println("DBG: Got bad message trailer.");        
                            //
                            stateRx = ST_HDR;
                            nSegIdx = 0;
                            nIdx = 0;
                            //
                            DebugPrintPacket();
                                                    
                        }//if
                        
                    }//if
                    else
                    {
                        bTrailerGood = false;
                        
                    }//else

                    nSegIdx++;
                                    
                break;
        
            }//switch
            
        }//while
        
    }//if
    
}//RxStateMachine

void DebugPrintPacket( void )
{
    //header
    Serial.print( "HDR... " );
    Serial.print( Packet.sPacket.grbHeader[0], HEX );
    Serial.print( Packet.sPacket.grbHeader[1], HEX );
    Serial.print( Packet.sPacket.grbHeader[2], HEX );
    Serial.println( Packet.sPacket.grbHeader[3], HEX );

    //payload
    Serial.print( "DATA1:\n" );
    for( int i=0; i<500; i++ )
        Serial.println( Packet.sPacket.grnData1[i], DEC );
    Serial.print( "DATA2:\n" );
    for( int i=0; i<176; i++ )
        Serial.println( Packet.sPacket.grnData2[i], DEC );
    Serial.print( "DATA3:\n" );
    for( int i=0; i<400; i++ )
        Serial.println( Packet.sPacket.grnData3[i], DEC );
    Serial.print( "DATA4:\n" );
    for( int i=0; i<276; i++ )
        Serial.println( Packet.sPacket.grnData4[i], DEC );
    Serial.print( "DATA5:\n" );
    for( int i=0; i<208; i++ )
        Serial.println( Packet.sPacket.grnData5[i], DEC );

    Serial.print( "DATA6 : " );
    Serial.println( Packet.sPacket.fData6, 3 );
    Serial.print( "DATA7 : " );
    Serial.println( Packet.sPacket.fData7, 3 );
    Serial.print( "DATA8 : " );
    Serial.println( Packet.sPacket.fData8, 3 );
    Serial.print( "DATA9 : " );
    Serial.println( Packet.sPacket.fData9, 3 );
    Serial.print( "DATA10: " );
    Serial.println( Packet.sPacket.fData10, 3 );
    Serial.print( "DATA11:" );
    Serial.println( Packet.sPacket.fData11, 3 );
    Serial.print( "DATA12:" );
    Serial.println( Packet.sPacket.fData12, 3 );
    
    Serial.print( "TRLR.. " );
    Serial.print( Packet.sPacket.grbTrailer[0], HEX );
    Serial.print( Packet.sPacket.grbTrailer[1], HEX );
    Serial.print( Packet.sPacket.grbTrailer[2], HEX );
    Serial.println( Packet.sPacket.grbTrailer[3], HEX );

}//DebugPrintPacket
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 24, 2019, 05:59 am
What exactly is the project?

What Arduino are you using?

Maybe a RaspberryPi would be more suitable?

...R
@Robin2,

I have a prototype sensor whose firmware has been set up to generate a bunch of raw debug output.  I need to setup a data logger to capture this output with the sensor placed in real-world conditions and then send the data over a cellular connection to be logged.  The data will be used to fine tune algorithms associated with the sensor.

I am using the Arduino Due.

A RaspberryPi might have been more suitable in terms of dealing with the amount of data to be transferred, however I'll be dealing with a very tight power budget.  I also don't really need all the overhead that the Pi brings with it.

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Juraj on Sep 24, 2019, 06:24 am
has the device a flow control pin?
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 24, 2019, 06:26 am
has the device a flow control pin?
No, it does not.  It implements TX @ 3.3V signal level, VCC @ 5VDC and GND.  It has a RX pin, however that is only used to change debugging modes on the prototype.  No handshaking possible.

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 24, 2019, 06:39 am
I've changed the indexing a bit and added a debug print at the end. How's this:

...
@Blackfin BINGO!  With debugging enabled it gets a bad tail every few passes, however with debugging disabled it flies right through every pass without a glitch.  I think the print's are slowing it enough that it's missing some characters in the RX buffer or something.  Since its aligning properly, I don't need the debugs now anyhow, so that's pretty moot!

The only thing I had to change to get it running was to modify the following lines to read as shown:

Code: [Select]

    if( Serial1.available() > 0 )
    {
        while( Serial1.available() )
        {
            cByte = Serial1.read();


I can finally take time now to wrap my head around the other subsystems.  I suspect now that RingBuffer.h has been returned to its default state the I2C libraries should play nice with the rest of my code.

I appreciate the advice and code examples greatly.  I'll let you know how the whole project turns out.

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Robin2 on Sep 24, 2019, 09:24 am
The data will be used to fine tune algorithms associated with the sensor.
That suggests that the firmware on the sensor can be modified. If so, re-write the debug-data-sending to be more Arduino friendly - for example send the data in manageable chunks.

I have no experience with a DUE.

...R
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 24, 2019, 06:26 pm
@Robin2

If I had access to the firmware, I would do that.  The prototype sensor is being developed by another company.  I plan to use it eventually, and it is being optimized for the use I have for it, but ultimately the firmware is outside my control.

The Due has a lot of SRAM and flash available, however I'm finding that some libraries are a bit dodgy on it.  It represents a fairly small sliver of the Arduino ecosystem, so I don't think it has had the degree of peer review the other boards have had.

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Paul_KD7HB on Sep 24, 2019, 07:27 pm
Have you determined that your program can process the data faster than it arrives?

If this was my project, I would program a circular buffer to store the data bytes taken from the serial buffer and when the end of the message is reached, begin to process the message data. The circular buffer may need to be 2X your message length.

Paul
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 24, 2019, 07:32 pm
@Paul_KD7HB Yes, the state machine implementation that @Blackfin suggested and provided code for is plenty fast enough to capture 3156 byte data packets, and that has been verified. 

I'll do the processing to get a complete packet sent to the modem and SD card before going back to the state machine to grab another packet.  There will be a few missed packets while I'm sending to the modem and SD-CARD, however that is not critical.  I only need the 10 packets collected each 10 minute interval to be close to each other temporally, they do not have to be sequential.

Rob
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Paul_KD7HB on Sep 24, 2019, 07:40 pm
This is your FIRST mention of writing the data to an SD card. Have you done a timing test to see how long it takes to write your data to an SD card? Pretty important test!

Paul
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 26, 2019, 01:11 am
This is your FIRST mention of writing the data to an SD card. Have you done a timing test to see how long it takes to write your data to an SD card? Pretty important test!

Paul
Title: Re: Proper method of receiving UART packet data far exceeding RX buffer size?
Post by: Rob_Mohr on Sep 26, 2019, 02:20 am
@Paul_KD7HB,

Yes, I've created code for all of the sub-components.  Time to write to the SD-card is not critical as it can occur after reading the data from the sensor, and after having turned the sensor and the modem off, thereby conserving power as the SD-card can take a bit by itself.

Rob