Reading large Amount of Data At high Baud Rate

Hi All,

I have had some great help here recently so I was hoping to pick your collective brains again if possible! Again I must stress Im very new to this so please bare with me.

I am trying to extract data from an automotive ECU, which uses RS232 comms, at a baud rate of 230400. In order to get the ECU to start reporting, you must send it a 'start report' request, and then a keep awake every ~2s. This is fine, and I have this working - the ECU begins reporting fine.

The length of one complete message back from the ECU (which contains all engine running data) is 549 bytes, and is sent over and over.

All I'm trying to do presently, is read the payload, and print specific bytes to serial monitor as a proof of concept eg byte 110 = engineRPM.

The first first three bytes of the payload are fixed at 0x4D, 0x45, 0x1C, so I'm using these to get the start of a message. This part of the code works consistently - when the start of a new message is received, I start to run through the rest of the bytes in the message.

The final byte in a message is variable in real life, but at the minute im using a fixed message for testing so I've defined it as 0x5F.

So... the problem I have is I can read the incoming message perfectly, but only up to 66 bytes. Then it seems there is nothing left to read.. if I was going to guess Id say the buffer has filled quicker than I can read through it - but if this is the case Im not sure how to work around it.

In terms of Hardware, i'm using a teensy 4.1 board, with a MAX3232 adapter to connect to the ECU. If its helpfull I can also share the fixed message Im using for testing?

My code is as follows:

char syncChars[2] = {'M', 'E',};
boolean newData = false;
const int numChars = 549;
char rdbyte[numChars];

byte req [] = { 0x4D, 0x45, 0x01, 0x00, 0x00, 0x00, 0x02, 0x01, 0x03, 0x05, };

void setup() {
  Serial1.begin(230400);
  Serial.begin(230400);
  Serial.println();
  Serial.println("Serial debug line open");
  Serial.println("Initialized serial");
  delay (6000);
  start_reporting_req();
}

void loop() {
  if (Serial1.find(syncChars)) { // Looking for bytes 0x4D , 0x45 to signal start of message
    delay(10);
    recvWithStartEndMarkers() ; // If I find the start of the message, I go to start reading the full message
    newData = false;
  } else {
    //Serial.println("timeout"); Nothing to see here!
  }
}

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  int ndx = 0;
  byte startMarker = 0x1C;    // this is third byte in the start of a message
  byte endMarker = 0x5F;      // this is what im using currently to singal the end of a message during testing
  byte rc;

  while (Serial1.available() > 0 && newData == false) {
    rc = Serial1.read();
    Serial.println(rc);

    if (recvInProgress == true) {
      if (rc != endMarker) {
        rdbyte[ndx] = rc;
        ndx++;
        Serial.print ("Byte number recieved=  "); Serial.println(ndx);  // This is what Ive added in for debugging - the max ndx value I see is 65
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else {
        Serial.print("Final NX Value =  ");
        Serial.println(ndx);
        rdbyte[ndx] = '\0'; // terminate the string
        Serial.println("End Marker recieved");
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }
    else if (rc == startMarker) {
      Serial.println("New data in now");
      recvInProgress = true;
    }
  }
}

void start_reporting_req() {
  Serial.print("Sending: 0x4D, 0x45, 0x01, 0x00, 0x00, 0x00, 0x02, 0x01, 0x03, 0x05 ");
  Serial1.write(req, sizeof(req));
  Serial.println();
}

That looks like a serial buffer overflow waiting to happen.

Think - you're getting a character every 44 microseconds

Thanks for your reply, I am thinking the same to be honest, however without that delay, I seem to 'miss' the next byte in the sequence which is 0x1C when I start reading the buffer, and thus I don't read anymore as its deemed not the start of a message.

I can't help feeling the problem lies with my use of Serial.find somehow?

Kind Regards

Joe

Don't mix the recvWithStartEndMarkers() function with Serial1.find(). recvWithStartEndMarkers() can be modified to look for the three-byte start sequence, you then keep calling that function, waiting for newData to become true, at which point you parse out the data you need. Avoid delays, that will quickly lead to loss of data, as can printing large amounts of data to Serial without calling recvWithStartEndMarkers() periodically to empty the Serial1 receive buffer.

I can't help feeling it's more to do with the 230 character-period deadtime right after the Serial.find

Perhaps one may consider that receiving serial and parsing serial are 2 different things.

Is the MCU a ESP32?

here is an example were receiving serial and parsing serial are seperate functions

void fParseLIDAR_ReceivedSerial ( void * parameters )
{
  // distribute received LIDAR info
  String sTmp = "";
  sTmp.reserve ( 20 );
  String sMessage = "";
  sMessage.reserve ( StringBufferSize300 );
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtParseLIDAR_ReceivedSerial, pdTRUE, pdTRUE, portMAX_DELAY) ;
    xQueueReceive ( xQ_LIDAR_Display_INFO, &sMessage, QueueReceiveDelayTime );
    //        Serial.println ( sMessage );
    int commaIndex = sMessage.indexOf(',');
    sTmp.concat ( sMessage.substring(0, commaIndex) );
    sMessage.remove( 0, (commaIndex + 1) ); // chop off begining of message
    //    Serial.println ( sTmp );
    if ( sTmp == "!" )
    {
      xSemaphoreGive ( sema_LIDAR_OK );
      //  Display info from LIDAR
      sLIDAR_Display_Info = sMessage;
    }
    if ( sTmp == "$" )
    {
      xEventGroupSetBits( eg1, evtResetWatchDogVariables );
    }
    if ( sTmp == "#")
    {
      // Serial.println ( "#" );
      xSemaphoreTake( sema_LIDAR_Alarm, xSemaphoreTicksToWait );
      sLIDAR_Alarm_info = sMessage;
      xSemaphoreGive( sema_LIDAR_Alarm );
      xEventGroupSetBits( eg, evtfLIDAR_Alarm );
    }
    //    Serial.println ( "parse serial ok" );
    sTmp = "";
    sMessage = "";
    xSemaphoreGive( sema_ParseLIDAR_ReceivedSerial );
    //    Serial.print( "fParseReceivedSerial " );
    //    Serial.print(uxTaskGetStackHighWaterMark( NULL ));
    //    Serial.println();
    //    Serial.flush();
  }
  vTaskDelete( NULL );
} // void fParseReceivedSerial ( void * parameters )
/////////////////////////////////////////////////////////////////////////////////////////////////////
void fReceiveSerial_LIDAR( void * parameters  )
{
  char OneChar;
  char *str;
  str = (char *)ps_calloc(300, sizeof(char) ); // put str buffer into PSRAM
  bool BeginSentence = false;
  sSerial.reserve ( StringBufferSize300 );
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( LIDARSerial.available() >= 1 )
    {
      while ( LIDARSerial.available() )
      {
        OneChar = LIDARSerial.read();
        if ( BeginSentence )
        {
          if ( OneChar == ‘>’)
          {
            if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
            {
              xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &sSerial );
              xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
              //
            }
            BeginSentence = false;
            break;
          }
          sSerial.concat ( OneChar );
        }
        else
        {
          if ( OneChar == ‘<’ )
          {
            sSerial = “”; // clear string buffer
            BeginSentence = true; // found begining of sentence
          }
        }
      } //  while ( LIDARSerial.available() )
    } //if ( LIDARSerial.available() >= 1 )
    xSemaphoreGive( sema_ReceiveSerial_LIDAR );
  }
  vTaskDelete( NULL );
} //void fReceiveSerial_LIDAR( void * parameters  )

Thank you for your response, I would like to go this route but Im ashamed to say I'm struggling a bit. Would you be able to provide an example f the modification? Sorry to ask what is probably a basic question!

Kind Regards

Joe

The other issue I see is trying to print more than you receive, at the same bit rate as you receive.
That's going to block very quickly.

Here is an example with multi-byte start markers, and a fixed length message. Have not fully tested it, only checked the receive function with a nano and the serial monitor, but should work with any arbitrary length of start marker.

You didn't mention much about the actual content of the payload, if it contains binary data you will not be able to search for a specific end marker, because the end marker might appear in part of the data.

const size_t numChars = 550;
char receivedChars[numChars];

boolean newData = false;

void setup() {
  Serial.begin(230400);
  Serial1.begin(230400);
  Serial.println();
  Serial.println("Serial debug line open");
  Serial.println("Initialized serial");
  delay (6000);
  start_reporting_req();
}

void loop() {
  recvWithStartMarkersFixedLength();
  if (newData == true) {
    Serial.println("Data received");
    //process data
    newData = false;
  }
}

void recvWithStartMarkersFixedLength() {
  static size_t ndx = 0;
  static size_t ndxStartMarker = 0;
  static const char startMarkers[] = {0x4D, 0x45, 0x1C};
  const size_t sizeStartMarker = sizeof(startMarkers) / sizeof(startMarkers[0]);
  //const char endMarker = 0x5F;
  char rc = '\0';
  enum {
    waitingForStartMarker,
    detectingStartMarker,
    receiveInProgress
  } static state = waitingForStartMarker;
  static bool reprocess = false;

  while (reprocess || ((Serial1.available() > 0) && (newData == false))) {
    if (!reprocess) {
      rc = Serial1.read();
    }
    reprocess = false;
    switch (state) {
      case waitingForStartMarker:
        if (rc == startMarkers[0]) {
          receivedChars[ndx++] = rc;
          ndxStartMarker++;
          state = detectingStartMarker;
        }
        break;
      case detectingStartMarker:
        if (rc == startMarkers[ndxStartMarker]) {
          receivedChars[ndx++] = rc;
          ndxStartMarker++;
          if (ndxStartMarker >= sizeStartMarker) {
            //complete start marker detected
            ndxStartMarker = 0;
            state = receiveInProgress;
          }
        } else {
          //start marker not detected
          ndx = 0;
          ndxStartMarker = 0;
          reprocess = true;
          state = waitingForStartMarker;
        }
        break;
      case receiveInProgress:
        //if (rc == endMarker) {
        if (ndx == 548) { //fixed length message
          //store end marker and terminating null
          receivedChars[ndx++] = rc;
          receivedChars[ndx] = '\0';
          ndx = 0;
          newData = true;
          state = waitingForStartMarker;
        } else {
          receivedChars[ndx++] = rc;
          //leave space in buffer for end marker and terminating null
          if (ndx >= (numChars - 1)) {
            ndx = numChars - 2;
          }
        }
        break;
    }
  }
}

const char req[] = { 0x4D, 0x45, 0x01, 0x00, 0x00, 0x00, 0x02, 0x01, 0x03, 0x05, };

void start_reporting_req() {
  Serial.print("Sending: 0x4D, 0x45, 0x01, 0x00, 0x00, 0x00, 0x02, 0x01, 0x03, 0x05 ");
  Serial1.write(req, sizeof(req));
  Serial.println();
}

Hi @anon56112670,

Thats amazing, thank you for taking the time to do that. I haven't been able to test it yet, but what you have done makes perfect sense, and has helped y understanding immensely, which is just as good as the code working!

I will run this as soon as I can and let you know how it goes, thanks again.

Kind Regards

Joe