Sending command over serial >64 bytes (>128 bytes)

Hello,

To elucidate the bracket in the subject: The description of the serial.available function says that the serial receive buffer holds 64 bytes, whereas most of the topics in this forum I browsed speak of 128 bytes. The datasheet of the Arduino Uno I am using doesn't say anything about it.. I just found out that I can send commands up to 70 bytes...??

The overview of my system:

I need to control several LED-Strips which each contain 32 LEDs over SPI with the Uno. In total I shall control as much LED strips as I can with 1 Arduino Uno. I designed a protocol communicating with the arduino over Hterm running on a Win 7 32-Bit system, which I store in a char Array. This protocol has a fixed part in total + 3*LEDs, whose R,G,B Data I want to change.

For example controlling 80 LEDs would be a fixed part of 16 bytes + 240 bytes LED-RGB-Data. As I already mentioned I found out, that controlling 18 LEDs is possible, making a command length of 16 bytes + 54 bytes =70 bytes. If I add one LED (=73 bytes in total) the 71st and the following bytes are -1. So first I am quite curios, why 70 bytes, and not 64 or 128?

Second, which is more important, is there any way to send a command with more than 64 bytes??

Thank you very much for your help!

Kai

Do you mean receive more than 64 bytes? As long as you pull the data out of the buffer quickly you can receive as much as you want.

This is an excerpt from hardware/arduino/cores/arduino/HardwareSerial.cpp (Arduino 1.0.1):

// Define constants and variables for buffering incoming serial data.  We're
// using a ring buffer (I think), in which head is the index of the location
// to which to write the next incoming character and tail is the index of the
// location from which to read.
#if (RAMEND < 1000)
  #define SERIAL_BUFFER_SIZE 16
#else
  #define SERIAL_BUFFER_SIZE 64
#endif

Serial buffer size is not something you find on the mcu datasheet… it’s just a software thing. But as Nick said, you shouldn’t worry too much about buffer size as long as you keep read()ing data off of it.

Suppose you have an end-of-message marker, for example ‘*’ (without the quotes). Let’s also suppose the largest message you expect is 100 bytes long. Then you’d have something like (pseudocode!):

const int BUFSIZE = 101; // 1 byte for c string terminator
const char EOM = '*';
char buffer[BUFSIZE];

while Serial.available() 
    ch = Serial.read()
    if ch == EOM then
        parseBuf(buffer); // message complete: parse it and take actions
    else
        if there's still room in the buffer
            store ch into buffer
        else
            // problem: incoming message is longer than expected...
        endif
    endif
end while

In all of this the serial buffer provided by Arduino is only used to store not-yet-read incoming bytes. Therefore, depending on how long it takes to parse and execute a message and the rate at which the messages come in, you should barley touch the 64-byte limit.

HTH

Ok, that is what I thought at first as well. So I just copied the relevant part of my program, where the buffer will be read in one by one in a unsigned char Array. I then deleted one line after the other to find the error and now I just have this basic rest of the code, but it is still not working…

unsigned char incoming_Char[230];


int a=0, b=0, c=0;
unsigned char Read_Serial=0;


void setup()
{
  Serial.begin(115200);

  
}

void loop ()
{
  a=0;

if (Serial.available()>0)
{
  while (Serial.peek() != 241 )  
  {
       Serial.print("reading: ");
       Serial.print(a,DEC);
       Serial.print("=");
       Serial.println(Serial.peek(),DEC);

       incoming_Char[a]=Serial.read();
   
       a++;
       Read_Serial=1;
   }
}

      if (Read_Serial==1)
      { 
        for (int a=0; a<PROTOCOL_SIZE; a++)
        {
        Serial.print(a,DEC);
        Serial.print(":");
        Serial.println(incoming_Char[a],DEC);
        }
        Read_Serial=0;
      }
}

As I want to set the brightness of the LEDs from 0 to 254 I am using Decimal as an input and am storing everything in the unsigned char array incoming_char.

With the specific input of "1 2 3 4 … 96 97 98 241 251 " (sent via HTerm 0.81) I get the following output


reading: 67=68<\r><\n>
reading: 68=69<\r><\n>
reading: 69=70<\r><\n>
reading: 70=71<\r><\n>
reading: 71=72<\r><\n>
reading: 72=77<\r><\n>
reading: 73=93<\r><\n>
reading: 74=-1<\r><\n>
reading: 75=-1<\r><\n>

?? You see, until 72 everything works fine, then there is a jump to 93 and in the following the serial buffer doesn"t seem to have a content at all (-1)…

Does anyone has an idea what is wrong?

… Thank you very much!

Kai

  • Serial.read() returns -1 when there's nothing to read. You either have to test for that or test for Serial.available() > 0 after each Serial.read() while you wait for the EOM.

  • Sending the ascii string "123" is not the same as sending the byte value 123.

  • PROTOCOL_SIZE is undefined, so you code doesn't compile.

You don't count how many characters you have received, and print the entire contents of the incoming_Char array. This way you'll see garbage or -1 after the last received char.

HTH

Thanks HTH for your reply.

*2: I am sending it as an DEC not ASCII (that is why the Serial.peek() returns the correct number and not its ascii code as you can see in the output)

*3: Unfortunately I forgot to change it during paste and copy. I used to have an array size of PROTOCOL.. but later I just changed to 230. So it should be 230 instead of PRO..

*4 I know that I am printing beyond the stored serial.read values. this is just for me to see whether at least the correct values are (magically) stored in the array, what they aren't of course. Actually you can leave that out as well, as the output above already shows the corrupt buffer values

*1 That is what I am talking about: There is no value anymore in the buffer, although I sent 100 values.. Why the jumps??..

As you are maybe not familiar with the syntax of the Serial lib:

a. Serial.read() returns the first value from the buffer and deletes it.

b. Serial.peek() just returns the first value.

So the output shows:

fixed string "reading"; the array element "a"; the actual value in the buffer (using Serial.peek)

Anyone any idea?..

Thanks!

Thanks HTH for your reply.

My nickname is tuxduino. HTH stands for "Hope This Helps" :-)

You need to test Serial.available() before each Serial.read() to be sure at least one (new) char has been received. Otherwise you have to take into account that Serial.read() might return -1, i.e. no (new) character was received at the time Serial.read() was called.

:roll_eyes: sorry tuxduino, didn"t know HTH :)

Of course you are right that during operation I should use Serial.available before each read. The problem here is just that there is nothing in the serial buffer, although I sent 1 2 ...97 241.

So my specific problem / question is why do I have a corrupt buffer values (jumps, which lead to a advanced end of the values stored in the buffer) indicated by the Serial.peek(). If that is solved I will include the if(Serial.available) {..=Serial.read;} then

Update:

Just doubled the software buffer values in the HardwareSerial.cpp (tuxduino was referring in his 1st post). Now it works.

As this can be intuitively ok for my main program, which e.g. has some if clauses during Serial.read(), I cannot understand why for the simple example I posted the buffer reaches values >68 (Especially because I also discovered, that changing the serial buffer iterativley to 9600 didn"t change anything). So am I already doing something "inefficient" in my example above (except from the print at the end, which doesn"t influence the buffer..). What bad impact (RAM Size, speed in general) does the enlarging of the buffer has?

I see doubling the buffer size as a (temporary workaround) so still would be very happy to have some ideas regarding my original problem.

If you rewrite your code by testing Serial.avalable() before read()ing the byte, I think you’ll also solve the corruption problem.

(edit: while it’s fascinating to mess with the core libraries, I’d leave this as an exercise to better understand the Arduino internals, not as a way to solve one’s own bugs :slight_smile: )

Yeah I know that it is probably not good to mess with the core libraries. That is why I see it as a temporary workaround.

Using Serial.available before reading the byte on the one hand indeed solves the corruption problem. But on the other hand, the cause for the corruption which I now identified as a “simple” buffer overflow is not solved.

So what I am really wondering about, and also would like to ask, is why does this simple reading byte program (you obviously cannot name this a program) already leads to a max. buffer content of 45 bytes when I send a 106 byte command (My goal is actually a 960 byte command…)? I mean, is there any inefficiency or, can I optimize the read process?

I attached the modified version, where the not necessary prints are deleted and a print of the serial.avalaible function (returning the bytes stored in the buffer) is added.

unsigned char incoming_Char[230];


int a=0, b=0, c=0;
unsigned char Read_Serial=0;


void setup()
{
  Serial.begin(115200);

  
}

void loop ()
{
  a=0;

if (Serial.available()>0)
{
  while (Serial.peek() != 241 )  
  {
       Serial.println(Serial.available(),DEC);

       incoming_Char[a]=Serial.read();
   
       a++;
       
   }
      Read_Serial=1;
}

      if (Read_Serial==1)
      { 
        for (int a=0; a<230; a++)
        {
        Serial.print(a,DEC);
        Serial.print(":");
        Serial.println(incoming_Char[a],DEC);
        }
        Read_Serial=0;
      }
}

Again, thanks for your help, it is very much appreciated!

(Hit CTRL+T in the IDE before posting, badly indented code is very hard to read!)

This piece of code:

while (Serial.peek() != 241 )
{
    Serial.println(Serial.available(),DEC);
    incoming_Char[a]=Serial.read();
    a++;
}

will happily overwrite poor Arduino’s RAM with -1, unless the byte 241 comes in in time. This is definitely not a robust piece of code. When you access an array element, you have to make sure the element index is less than the total number of array elements, otherwise you might read garbage, or worse you can overwrite portions of RAM which might belong to other part of the program, causing data corruption or (most likely) a program crash.

Here’s a sketch that will scale to whatever message length you want to use (provided you don’t use the entire RAM for a simple buffer).

const int BUFSIZE = 230;         // max message size is BUFSIZE - 1 to leave room for the end-of-string NULL character (i.e. 0x00)
const char EOM = '*';            // marks the end of the message

char buffer[BUFSIZE];    // here we store the message as we receive it char by char
int  bufPos = 0;                // where to store the next received char in buffer


// this function will have to analyze the message's contents and
// take appropriate action
void processMessage(const char *msg) {
    // this is only a test, so we just
    // print out the message
    Serial.print("Received: ");
    Serial.println(msg);
}


void setup()
{
    Serial.begin(115200);
}


void loop ()
{
    char ch;
    
    while (Serial.available() > 0) {
        ch = Serial.read();
        
        if (ch == EOM) {            // end-of-message
            buffer[bufPos] = 0;     // string terminator
            processMessage(buffer);  // do something with the received message
            bufPos = 0;             // restart for next messages
        }
        else {
            if (bufPos < (BUFSIZE - 1)) {        // if there's still room in the buffer
                buffer[bufPos] = ch;             // store the received char
                bufPos++;                        // forward 1 position in buffer
            }
            else {
                Serial.println("Lost byte!");    // no more room, have to throw away the received char
            }
        }
    }
}

Instruction: throw as many character as you want at the Arduino. It will store them inside buffer until it receives *. If * doesn’t come before the buffer is full, it will print out an error message. When * is received, it will print the buffer contents and start over, waiting for the next message.
This code works with messages of any length, up to BUFSIZE-1.

My goal is actually a 960 byte command…

Beware, you’re using half the RAM of Arduino just to store a message… You’ll quickly run into out-of-ram bugs. Rethink you design.

Hey,

again thanks tuxduino that you are trying to help. Didn't know the AutoFormat Function, pretty cool ;).

Unfortunately, you, tuxduino, are not really getting my point.As some others may not got it either I like to explain:

  1. With the piece of code I posted above, I will get corrupted values (with a lot of -1 at the ends) and an array overflow a.I should avoid that by only using the specified array size (What Tuxduino mentioned in his last post) b. I should avoid/filter them using an if clause (What Tuxduino mentioned in the posts before)

to 1a. The piece of code I posted is just a quick and dirty code to have a fast read as a kind of "performance maximum". Therefore no if clauses and also no control of the array borders (I am not a coding pro, but I know these basics ;) )

to 1b. Right now, I am trying to get the max. command length possible, so I don't care about the lots of -1. So with my piece of code, which does only the most basic to read the serial, having a command length of 106 bytes already leads to a max buffer content of 45 bytes.... Having a command length >150 already leads to a buffer overflow, bytes of the commands are dropped, the command itself is unusable (Of course I have some functions to check whether the command is transmitted completely)

That leads me to my actual question:

  1. either I am doing something wrong, so to say an inefficient way of reading or similar

  2. or the "reading the serial" performance is just too slow for my requirements (I already considered splitting the 960 byte cmd I want to 3*320 e.g.)

3.Does increasing the serial buffer has some negative impact? I guess the buffer gets its memory of the SRAM?

So I would like to have some opinions on these 3 points. You are all welcome to leave it here :)

Thanks!

I'm glad you appreciated my help, but I think we've come to a point where we just repeat ourselves.

Either you understand how to properly receive serial bytes, or you'll keep trying to debug the unfixable. I suggest you do a google search about exactly how serial transmission works, at the lowest level. You might eventually convince yourself that serial transmission speed doesn't increase if the receiver calls read() a million times per second...

Good luck.

PS: if (a >= BUFSIZE) { a = BUFSIZE; } takes sooooo long... :P

I am not working on it today anymore, but, to focus on the real problem I see I will integrate the array border check tomorrow.

@Tuxduino: Did you try to run your piece of code with a command length >120 bytes? I guess you will see some jumps, that why I did when I tried it this morning and am also experiencing with my piece of code. And that is the actual problem which I am referring to. The thing is chars/bytes of the command transmitted will be dropped if the command is around 110 bytes long because the buffer is at its limit (64 bytes) and is not able to store the following bytes until the first byte has been read. Therefore you see some jumps if you print out the array. And that is what I like to fix, but don't know how as its cause lies in the buffer processing.

@Tuxduino: Did you try to run your piece of code with a command length >120 bytes? I guess you will see some jumps

Of course I did.

What do you mean by "jumps" ? Can you run it again and post here both what you sent and what you got in response ?

change the 64 in this line of SoftwareSerial.h to 256 worked for me.

define _SS_MAX_RX_BUFF 64 // RX buffer size