How to receiver larger data using UART into a byte array

[mileslin]

Hi All,

I would like to receive data from the sensor to my esp32-cam, then save them into SD CARD.
The sensor has two kinds of packet formats.
One of them which is 1500 bytes in length is the target to receive.
It has specific headers and tails.

I tried to change the buffer size of RX still received less than 1500 bytes of data.
Hope anyone knows what's wrong.

#include "SD_MMC.h"
#define UART0_RX_SIZE     2048
byte recData[2048];
String filename;
void setup() {
  Serial.begin(115200);            
  Serial.setRxBufferSize(UART0_RX_SIZE);
  Serial.setTimeout(20);  

}

void loop() {
  if(Serial.available()>0)
  {
    Serial.readBytes(recData,sizeof(recData));
  }
  if(recData[0] == 0x51 && recData[1] ==0xAA)
  {
    String myString = String((char*)recData);
    appendFile(SD_MMC,filename,myString);
    memset(recData, 0, sizeof(recData));
  }

}

[330R]

What kind of data is that? It doesn't look like ASCII. Apart from the header 0x51, 0xaa, that you could discard (but aren't discarding), the image shows values like 0xfe and 0xf2: this is not ASCII either. Maybe it's some kind of extended ASCII?

Can you post an example of how this string is supposed to look like, given a certain input?

[Koepel]

Get rid of the .readBytes(), it has a default timeout of 1 second, and it is hard to synchronize with incoming data. If it stops because of a timeout, do you know if the data has a zero-terminator ? I don't.

The gap between the serial data is only small and the Arduino functions are not meant to detect that gap.

Can you tell more about the format ? Does it always start with 0x51, 0xAA and end with 0x0D, 0x0A ? Then you can ignore the gap and detect the begin and end.

[mileslin]

Actually, my esp32-cam is connected with the UART port on MCU, the data is a combination of many parts.
For example like this.

{0x51,0xAA,......,0x0D,0x0A,0x52,0xAA,......,0x0D,0x0A,........,0x0D,0x0A,.......,0x55,0xAA,....,0x0D,0x0A}

I could not judge from the tail!

[red_car]

Does it always start with 0x51?

Is it always 1500 bytes?

[mileslin]

Does it always start with 0x51? Correct!

Is it always 1500 bytes? Yes!

[red_car]

You could try something like this...

uint16_t  idx;
byte b;
const uint16_t RECORD_SIZE = 1500;
byte buffer[RECORD_SIZE];
boolean started = false;

void setup()
{
  Serial.begin(115200);
}

void loop()
{

  while (Serial.available())
  {
    b = Serial.read();

    if (b == 0x51) // Start byte
    {
      idx = 0;
      started = true;
    }

    if (started) // Ignore anything received until we get a start byte.
    {
      buffer[idx++] = b;

      if (idx > RECORD_SIZE - 1)
      {
        
        // Process the data here
        
        started = false;
      }
    }
  }
}

[cotestatnt]

@mileslin

Serial.setTimeout(20);

Why this instruction? 1500 bytes at 115200bps take longer time than 20ms (at least 105ms) as you can see also in your plot.

[sterretje]

@cotestatnt

1500 bytes at 115200bps take longer time than 20ms

But a single byte doesn't; each time that the code waits for the next byte, the timeout 'counting' starts again.

[cotestatnt]

Are you shure about that? I've checked, it is.
I have to admit, I've never used timeout with an async communication (it's a non sense from my point of view especcially if you have a "start" and a "stop" bytes).

Anyway, I think it would be a better approach to read byte by byte up to both terminators bytes which is always 0x0D, 0x0A (carriage return, new line), instead of reading a fixed number of bytes.

[Koepel]

Yes, in this situation it is better read the bytes one by one and deal with the protocol in the code, such as red_car gave a start in reply #7.

In my opinion this is always better than .readBytes() because of its undocumented behavior. The disadvantage is that you have to use millis() to be able to detect a timeout. With a timeout the code gets larger, see my millis_serial_timeout.ino

[sterretje]

@Koepel

because of its undocumented behavior

You know what programmers always say: "the code is the documentation"

[cotestatnt]

From my point of view the real issue, if a timeout occurs during communication, is that I would handle it on the sketch side in order to take some actions related to the application.

The Stream class seems missing feedback about timeout, it simply exit the while loop.

[Blackfin]

@mileslin

0x51,0xAA,......,0x0D,0x0A,0x52,0xAA,......,0x0D,0x0A...

The second packet in the sequence you posted seems to start with 0x52; or am I missing something?

You also said:

receive data from the sensor to my esp32-cam ... esp32-cam is connected with the UART port on MCU...

Can you clarify your setup: You say you want to "receive data from the sensor to my esp32-cam"; what sensor is this? Or is this just the esp32-cam sending serial data to an Arduino board?

You said

[mileslin]

This is a developed product that combines many sensors.

Originally used a computer to record data and analysis, but I have to record sensor data in esp32-cam for special applications.

[mileslin]

Hi all,
I have solved this problem using the string length method.
But it could not exactly receive 1500 bytes, so have to check the data length every time.
this work:

#include "SD_MMC.h"
String filename,recData;
const uint16_t RECORD_SIZE = 1500;

void setup() 
{
  Serial.begin(115200);            
  Serial.setTimeout(10);

   // Init SD_MMC
	Serial.print("Initializing SD card...");
	 if(!SD_MMC.begin("/sdcard", true)){
        Serial.println("Card Mount Failed");
        return;
    }
    uint8_t cardType = SD_MMC.cardType();

    if(cardType == CARD_NONE){
        Serial.println("No SD_MMC card attached");
        return;
    }
	Serial.println("Initialization done.");

	
    filename = "/001.dat";
    if(SD_MMC.exists(filename))
        SD_MMC.remove(filename);
    File newFile = SD_MMC.open(filename,FILE_WRITE);
    newFile.close();
    Serial.print("File & Folder setup completed!");

}

void loop() 
{
	if(Serial.available()>0)
	{
		recData = Serial.readString();
		if(recData.charAt(0) == char(0x51) && recData_s.charAt(1) == char(0xAA))
		{
			if(recData.length() > RECORD_SIZE)
			{
				recData.remove(RECORD_SIZE,recData.length()-RECORD_SIZE);
			}
			appendFile(SD_MMC,filename,recData);
		}
	}
}