Serial buffer limited to 64 bytes?

Projects Programming
1

Hello!

I have many Arduinos acting as transceivers (ESPs, Nanos, etc). They can all receive and transmit data at 433MHz through a module connected on SoftwareSerial pins.
Everything works fine on those. But the device with the XIAO fails to receive more than 64 bytes of data.
My project has thousands of lines of code but the relevant could be simplified (and be reproduced) as follows :

void loop() {
  while (rfModule.available()) {
    Serial.write(rfModule.read());
  }
}

where

SoftwareSerial rfModule(7, 6);

at baudrate 9600, for both software and hardware serials.

Below is what I read on the serial monitor on the XIAO when the XIAO is receiving SHORT data

<1OSKbXcOwsNeEiSJIikqrdNoMB1nnAD2NYJ9l5vAHrU=>

Below is what I read on the serial monitor on the XIAO when the XIAO is receiving LONGER data

<kweITGBlQfhqHSsZ1+BGO6DfwSlStRgrX2ZQKSkQsY5f+wTMj18TGldDcZPojK

as you can see, it is truncated.

(logs on transmitter shows data was fully sent, uncorrupted, and successfully received uncorrupted and full size on other devices such as an ESP8266, so the problem must be the XIAO).

chars < and > are delimiters for the encrypted data.

As far as I know, on any other device, method .read() will read one byte at a time and remove it from the buffer (whether hard/soft serial). I have no clue why this does not work on the XIAO.

Help would be appreciated.

(Device is Seeeduino XIAO SAMD21 Cortex M0)

2

Does your communication protocol include ANY error detection?

3

There is indeed a 64 byte buffer limit. I expect that other things in your code are taking enough time that the buffer overflows.

4

How is this related to my question/issue?
Define error detection.
Data is encrypted and contains a crc32 checksum to ensure everything I process is not corrupted.

5

I suspect a buffer overflow as well, but is this possible when I trimmed down the code that deals with the serial to basically this :

  while (rfModule.available()) {
    Serial.write(rfModule.read());
  }

I'm inside a while loop, so there is nothing else to process except reading a byte from the softwareserial (9600 bauds) and forwarding it to the hardware serial (9600 too).

6

Yes. The Arduino is fast enough that it will read the first character to arrive and then the next will still be on the wire. So your while loop will exit and the rest of your code will execute. If it takes too long before you get back to the while loop, there will be overflow.

7

Clear and concise explanation. Cheers.
Is there a documentation on how to tackle this elegantly?

I don't know how the available() behaves under the hood but in my mind it was as if it would return false only when the serial is not feeded a new byte AFTER SOME DELAY.
So right now I'm thinking about introducing some kind of delay so that it does not exit the while loop until everything has been received.

8

I have removed everything inside my loop() except this while() and I can confirm your explanation is absolutely accurate.
I never thought I was leaving this while() loop the whole time...

9

You can easily modify the source code for the Serial device to increase the buffer size. It is a simple #define in the source file. On Due's, I often increased the buffer to 256 bytes, enabling it to easily handle hundreds of often very long messages per minute at 115200 BAUD.

10

I tried that already when debugging.
The following never worked for me, even after modifying .h/.cpp source files directly :

#define SERIAL_TX_BUFFER_SIZE 128 // C:\Program Files (x86)\Arduino\hardware\arduino\avr\cores\arduino\HardwareSerial.h
#define SERIAL_RX_BUFFER_SIZE 128
#define _SS_MAX_RX_BUFF 128 // C:\Program Files (x86)\Arduino\hardware\arduino\avr\libraries\SoftwareSerial\src\SoftwareSerial.h
11

Basically, you have to get back to serial reading code quickly. So using delays is risky, reading DS18B20s synchronously is a disaster. At 9600 baud, each character takes about a millisecond on the wire. So you have about 64mS to start reading again to avoid overflow.

One way is to divide up your other functionality into small bits and only do one of them on each iteration of the loop function so you can check for serial activity more quickly.

It may be that you can read serial data until you see a '>'. Then you're wasting time waiting and you'll need a timeout to ensure you don't get stuck after some line noise corrupted the character you were looking for. But maybe that doesn't matter for your application.

12

Because you mention the delimiters you are using. If you encrypt the message that contains the delimiters, you must decrypt before seeing the delimiters. If you encrypt before adding the delimiters, there is the possibility that the encryption may create one of the delimiters.

Any data error may also create a delimiter.

13

The last paragraph is exactly what I was planning to implement.
However, I figured I was editing the wrong file. The XIAO SAMD21 is, as its name suggests, a SAMD, not AVR. I was editing the wrong source file. Here it is : C:\Users\MY_USERNAME\samd\1.8.3\libraries\SoftwareSerial\SoftwareSerial.h
I upgraded it to 128 instead of 64. I chose the easy path.
Thank you very much @wildbill

Something unrelated: you used the word "synchronously " about the DS18B20. I know from experience on an other project using it that the oneWire takes indeed about 750ms to obtain a reading. Is there a way to read asynchronously?

@Paul_KD7HB my encrypted data is base64 encoded. There are no chevrons in b64.
Yes, data error could create a delimiter, also a lightning strike could hit my chip, or my kid could spill water on the circuit. Still not sure why you were so focus on this...

14

Many, many years in financial institution data communication. Managing and writing software.

15

Yes. There's a function in the library to set it to asynchronous mode and then you have to manage the 750mS yourself but you can do other things while you're waiting for the sensor(s) to get a reading.

16

You can kick off the DS18B20 temperature measurement / conversion and then have your code do other stuff rather than blocking (doing nothing) for the 750ms. The DS18B20 will do its job without further intervention. Then later (sometime >=750ms), read the answer.

17

You have all been of great help. Thank you very much for your time. Moving forward now.
This thread is solved.