SoftwareSerial Rx buff increase to 65 (from 64) corrupts message...

Hi,

I am lost as to where to look next on an issue, and I can't find anything already discussed about this, so I am hoping someone can point out something really easy I have simply missed .
:slight_smile:
I have a project using SoftwareSerial and I need more than 64 byte buffers. It seemed like I just needed to change _SS_MAX_RX_BUFF to anything up to 254 and that would be that. However it seems to be corrupting the data somehow.

I have binary data coming in that always starts with DLS_4 (where the underscore represents a non-printing character). If I have :

#define _SS_MAX_RX_BUFF 64 // RX buffer size

in SoftwareSerial.h

then the data comes in as expected e.g. DLS4r (followed by varying data)

However, if I change the one line to

#define _SS_MAX_RX_BUFF 65 // RX buffer size

Then the data gets oddly corrupted e.g. Dj4Rªl1eY

The Dj4 is repeatable - every time it should say DLS4, so whatever is happening seems consistent.

I tried the following simple program as a test, and this is repeatable with any value I tried for the buffer size over 64. Back to 64 and it works fine again.

#include "SoftwareSerialnik.h" // Software serial port library copy with changed buffer size
#define V850RxPin 4     // Pin D4 for incoming V850 serial
#define MAX_WRITE_BYTES 200  //  Max number of bytes to write to file in one go.


SoftwareSerial V850Serial(V850RxPin, 3); // Software serial for V850 DLT. Rx pin is D4. Tx pin is not used, but set to D3 as we have to specify one. D3 is available on the Ardulog but not populated by JaNiX

// Variables for reading serial data
byte inByte[MAX_WRITE_BYTES]; // Temp buffer between serial and file
byte NumBytesToRead; // How many bytes to read at a time

void setup()
{
  //pinMode(V850RxPin, INPUT); // V850 softwareserial Rx pin - not sure this is needed


  //Setup serial ports
  V850Serial.begin(115200); // V850 in software serial
  Serial.begin(115200);

}

void loop()
{
  // Check and record data from V850
  if (V850Serial.available() > 0)
  {
    //NumBytesToRead = V850Serial.readBytes(inByte,  min(V850Serial.available(),  MAX_WRITE_BYTES)); // Read as many bytes as are available up to MAX_WRITE_BYTES
    NumBytesToRead = V850Serial.readBytes(inByte,  1); // Read one byte
    Serial.write(inByte, NumBytesToRead);
  }

}

"SoftwareSerialnik" is simply a local copy of SoftwareSerial that allows me to change the buffer size easily to experiment.

As per the commented-line in that test code, this seems to happen whether I read one byte at a time, or multiple bytes using readBytes. I also tried the simple V850Serial.read() and that also fails above 64 bytes.

Is there anywhere else I need to modify if I want to set a bigger Rx buffer?

Thanks in advance

Nick

I'd wager that some other place in the library isn't checking that define and is assuming it to be 64.

Do you really need a larger buffer?

As long as you read it periodically you are generally fine.

Receiving serial characters-especially at standard baud rates- are really slow compared to the arduino.

Hi,

I did a quick search around for "64" or other hard-coded values that might be relevant, but not found one yet, but I shall keep looking!

As to the need, I am actually logging 2 channels, one via HW serial and one via SoftwareSerial. Most of the messages are pretty short, but the occasional one will fill the buffer on it's own. I can set a much larger buffer on the HW serial and service the SoftwareSerial more often, and I am trying to juggle the buffer sizes to help. But, I want to have the headroom in case I need it.

Plus, a bit of me wants to understand what is going on :slight_smile:

Nick

As to the need, I am actually logging 2 channels, one via HW serial and one via SoftwareSerial. Most of the messages are pretty short, but the occasional one will fill the buffer on it's own. I can set a much larger buffer on the HW serial and service the SoftwareSerial more often, and I am trying to juggle the buffer sizes to help. But, I want to have the headroom in case I need it.

You can deal with a dozen or more serials (rx) each taking messages of giga bytes each. And you can do it all at the same time with much smaller buffers.

If you ever see available report more than 4 chars in the buffer than you are doing somthing VERY VERY badly wrong in the rest of your program.

Mark

Hi,

I am writing both serial ports to files on an SD card, which is where my slow-downs come from. I need to cater for a spike in data just as I am flushing the data to files etc.

I shall monitor the buffer sizes and throw what I can at it, and see what happens! Then I shall know what to optimise.

Oddly, lots of other people seem to just change the buffer size with no issue, which makes me wonder what I am missing, especially in that short test program...

Nick

The SoftwareSerial code uses arrays and a modulo function to do the queuing and wrapping.
I've implemented this type of head/tail queueing many times and I've never implemented it that way.
The issue you may be running into is that if the buffer size is a power of 2 the modulo function will optimize into an AND instruction. If it is not, then it will turn into a real divide which can take a lot longer.
I'm guessing that this may be what is corrupting your data.
For a quick test you could compare results of using a buffer size of 128 vs 129.
If 128 works and 129 fails, you'll need to go in an fix their code to do the wrapping with some better code.
There is no need to use a modulo function as there are better ways to do the same thing.

--- bill

1 Like

Hi Bill,

Many thanks - that did it :slight_smile: 128 works and 129 fails. I'll have a look at the code and see if I can spot anything that my limited skills can fix.

With the added benefit of hindsight, this explains why I thought I had got it working before. My first buffer increase was simply to double it from 64 to 128, conveniently sidestepping the issue. I only noticed the corruption later on, after I had tweaked a load of things including changing the 128 to 200.

Thanks again,

Nick

I suspected as much, particularly since you are using a high baud rate.
So that means it is likely a timing issue caused by the added latency of the modulo calculation.
The code is using a modulo calculation in the direct real time path of receiving bits in the ISR.
It would likely work if the baud rate were a bit slower.

I would trash all the queuing/de-queuing code and re-write it for smaller/better/faster code.
However, for your environment, you can probably get away with a small change:
(I have not tested this code to verify it)
In the function recv()
Change this:

    // if buffer full, set the overflow flag and return
    uint8_t next = (_receive_buffer_tail + 1) % _SS_MAX_RX_BUFF;

To this:

    // if buffer full, set the overflow flag and return
    uint8_t next = _receive_buffer_tail +1;
    if( next >= _SS_MAX_RX_BUFF)
        next = 0;

--- bill

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data.

You can receive as much data as you want without interfering with the SoftwareSerial buffer size. In the demo the receiving array is set to 32 characters but you can easily change that.

...R

Thanks Bill - I'll give that a try.

Hi Robin2 - I read through that thread before (and the pre-cleanup version first before I found that one!!). I have 2 115200 Serial streams coming in (one HW, one SW), and logging to SD card, and I am not in control of the content of either. The messages can be any length, so I can't pre-empt anything.

I originally tried with single byte reads, but, with writing to the SD card and also wanting some headroom left for analysis, I was not emptying the buffers quickly enough (not sure if that was just poor code on my side or asking too much of the HW). Switching to multi-byte reads made it easier to chunk up my SD-writing. So, I thought I would set some big buffers to handle any peaks, and that is when I noticed this.

Thanks for the pointers, though.

Nick

nrobbert:
(not sure if that was just poor code on my side or asking too much of the HW

Without seeing your code ....

I can't think of any reason why the performance of the system in my examples would be noticeably different from changing the buffer size. An Arduino can empty the Serial input buffer very much faster than data can arrive - even at 500,000 baud.

How are you getting SoftwareSerial to work at 115200 baud? I though 38400 was about the max and that would be pushing it. SoftwareSerial places a heavy computational load on an Arduino.

...R

Robin2:
Without seeing your code ....

I can't think of any reason why the performance of the system in my examples would be noticeably different from changing the buffer size. An Arduino can empty the Serial input buffer very much faster than data can arrive - even at 500,000 baud.

I just mentioned a potential one for SoftwareSerial which is what Nick is using and having issues with.
When the buffer queue wrapping is done using a modulo function instead of simple compares, you end up with division instead of AND instructions for non power of 2 buffer sizes, which take considerably longer.
That will add extra overhead.

So using buffers that are not modulo two makes the code larger and slower - this includes the recv() routine which is called from an ISR.
This extra overhead might cause issues.
(It would take some real analysis using some profiling with a logic analyzer to know for sure)

It would be interesting to look at the actual assembler output of recv() to see how it changes.

In the bigger picture, Async serial communication is not a reliable transport and applications should not falsely make the assumption that it is.
The best thing to do would be to use a bi-directional protocol over the link to make it reliable.
However, in this case from Nick's comment it sounds like it is not possible to alter the code on the transmitter.

--- bill

1 Like