Unexpected behaviour of Serial.available after Serial.read

So I was experimenting with Serial to read large chunks of bytes and I came across this behavior with Serial I wasn't expecting.

Referring to the following code:

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (Serial.available())
  {
    char ch = Serial.read();
    while (Serial.available())
    {
        char p = Serial.read();
        Serial.println(p);
    }
  }
}

By giving an input of say "abc", I was expecting the Serial monitor print "a", "b", "c" on each line. However, nothing were printed. Strangely, if I were to comment out the following line:

char ch = Serial.read();

It works as expected!

Looking at the way Serial.read() is implemented, it seems like the tail pointer gets incremented by 1. However, this shouldn't return false from Serial.available() in this case since reading "a" will leave with "b" and "c". I think I am missing something about understanding how Serial buffer works. Does anyone knows what is going on that explains this behavior?

Maybe you should re-read what available () and read () return.

the CPU is much faster then Serial

nothing is available.
'a' arrives,
1 byte is available and is read.
nothing more is available.
'b' arrives,
1 byte is available, and is read.
nothing more is available.
'c' arrives,
1 byte is available, and is read.
nothing more is available.

without the first read():

nothing is available.
'a' arrives, 1 byte is available
still is 1 byte available.
'a' is read and printed
nothing more is available.
'b' arrives, 1 byte is available
still is 1 byte available.
'b' is read and printed
nothing more is available.
'c' arrives, 1 byte is available
still is 1 byte available.
'c' is read and printed
nothing more is available.

Juraj:
the CPU is much faster then Serial

nothing is available.
'a' arrives,
1 byte is available and is read.
nothing more is available.
'b' arrives,
1 byte is available, and is read.
nothing more is available.
'c' arrives,
1 byte is available, and is read.
nothing more is available.

without the first read():

nothing is available.
'a' arrives, 1 byte is available
still is 1 byte available.
'a' is read and printed
nothing more is available.
'b' arrives, 1 byte is available
still is 1 byte available.
'b' is read and printed
nothing more is available.
'c' arrives, 1 byte is available
still is 1 byte available.
'c' is read and printed
nothing more is available.

Ok that makes so much sense now. Thank you for pointing that out!

Have a look at the examples in Serial Input Basics - simple reliable non-blocking ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

...R

Robin2:
Have a look at the examples in Serial Input Basics - simple reliable non-blocking ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

...R

Thanks for linking that! I should have pointed out earlier that I was using those examples but tweaking them to better suit my project needs and thus came across this behavior I wasn't expecting.

lekok:
using those examples but tweaking them to better suit my project needs

I don't see any evidence of the functions from my tutorial in the code in your Original Post

...R

Robin2:
I don't see any evidence of the functions from my tutorial in the code in your Original Post

...R

I have extracted out the portions I thought it was irrelevant to original question. But for completeness, the following is what I had:

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (Serial.available())
  {
    byte bite = Serial.read();

    if (bite == BEGIN_CHAR)
    {
      current_len = 0;
      clear(); // Zeroes the buffer array.
      while (Serial.available())
      {
         if (current_len >= SERIAL_READ_SIZE) // Maximum allowable size of one batch of data.
         {
           break;
         }
         byte next = Serial.read();
         buffer[current_len++] = (uint8_t)next;
      }

      // Process the buffer further down
    }
  }
}

Again, I have extracted out other parts, the entire sketch is pretty large with multiple components.

What I'm trying to achieve with the above is to look for the start marker and read a specific number of bytes and ignore any additional bytes until the next start marker is seen.

Basically what I am trying to achieve for my project:

  • Read large chunks of data from Serial.
  • Transmit these chunks using this Grove LoRa radio through a mesh network using the RadioHead library.

I have decent progress with 2. but 1. is where I am at.

Aside from that, I have also came across another one of your post where you shared a code that reads bytes that can consist of the start/end marker value, using a special marker.

However, your example assumes that the sender adheres to a certain rules when sending those bytes. That is marking payload bytes that is start/end marker by putting the special marker in front. However, in our use case, that may not be possible because the data is large (could be images) and has to be quickly processed on the sender's side. So I am looking for alternatives that is simpler and requires less computation.

So, I am wondering if there is any technique that can account for the following specifications:

  • The payload bytes can vary in sizes.
  • The payload data can be start or end markers and there could be multiple of them.

Example 3 in the link you gave will work well except for the concern of how would I know when the payload bytes has ended.

Thank you for your time so far.

You have a while loop inside the "if (Serial.available ()...," block.

Unless "clear()" takes a significant amount of time, that while will never even start.

TheMemberFormerlyKnownAsAWOL:
You have a while loop inside the "if (Serial.available ()...," block.

Unless "clear()" takes a significant amount of time, that while will never even start.

Right, it brings back to what Juraj mentioned earlier right? If the next byte has yet to arrive at the buffer, the MCU would have reached the while loop and Serial.available() would return 0 since it so much faster than Serial.

In that case, would the general rule of thumb be, to not read() any before I do a available()?

The rule is never do a read unless you've checked availability.

Use a state machine.

TheMemberFormerlyKnownAsAWOL:
The rule is never do a read unless you've checked availability.

Use a state machine.

Ok, I will keep this in mind. Thanks for these information, I very much appreciate them. :slight_smile:

lekok:
What I'm trying to achieve with the above is to look for the start marker and read a specific number of bytes and ignore any additional bytes until the next start marker is seen.

This came up in another Thread recently and this is what I suggested.

...R

PS ... its much easier to help if you stay as close as possible to the original version of a piece of code - so the changes you have made are obvious within a body of familiar code

You could also use a proven and efficient serial transfer library to send standardized byte packets between Arduinos. The library uses start/end markers plust cyclic redundancy checking, consistent overhead byte stuffing, and dynamic payload length handling. I use it for several of my own projects and should work perfectly for this one, too.

You can install it using the Arduino IDE's Libraries Manager (search "serialTransfer")

Example code:

TX Arduino:

#include "SerialTransfer.h"

SerialTransfer myTransfer;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);
  myTransfer.begin(Serial1);
}

void loop()
{
  float myFloat = 100.5;
  
  myTransfer.txBuff[0] = 'h';
  myTransfer.txBuff[1] = 'i';
  myTransfer.txBuff[2] = '\n';
  myTransfer.txFloat(myFloat, 3); //insert the float "myFloat" at index 3 since "hi\n" already takes up indicies 0-2
  
  myTransfer.sendData(7); //3 bytes for "hi\n" plus 4 bytes for the float "myFloat"
  delay(100);
}

RX Arduino:

#include "SerialTransfer.h"

SerialTransfer myTransfer;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);
  myTransfer.begin(Serial1);
}

void loop()
{
  float myFloat;
  
  if(myTransfer.available())
  {
    /////////////////////////////////////////////////////////////// Handle Entire Packet
    Serial.println("New Data");
    for(byte i = 0; i < myTransfer.bytesRead; i++)
      Serial.write(myTransfer.rxBuff[i]);
    Serial.println();

    /////////////////////////////////////////////////////////////// Parse Out Float From Packet
    myTransfer.rxFloat(myFloat, 3);
    Serial.print("Received float: "); Serial.println(myFloat);
    Serial.println();
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

Robin2:
This came up in another Thread recently and this is what I suggested.

...R

Noted this.
What if I want to allow varied payload size? I thinking if the sender sends something like: ><123>
Where '<' and '>' are start/end markers respectively. I wouldn't know if the data ended after the 'C' or '3' since the markers themselves can appear in the data and there is no fixed length to read.

As for the special marker method you posted, I'm concerned with having too much processing done on the sender and receiver side.
I'm thinking of using headers to inform the receiver how many bytes to receive but this will incur some overhead. I guess this is the trade off I have to compromise for.

PS ... its much easier to help if you stay as close as possible to the original version of a piece of code - so the changes you have made are obvious within a body of familiar code

That makes sense, I will keep this in mind when I post in future.

Power_Broker:
You could also use a proven and efficient serial transfer library to send standardized byte packets between Arduinos. The library uses start/end markers plust cyclic redundancy checking, consistent overhead byte stuffing, and dynamic payload length handling. I use it for several of my own projects and should work perfectly for this one, too.

You can install it using the Arduino IDE's Libraries Manager (search "serialTransfer")

Example code:

TX Arduino:

#include "SerialTransfer.h"

SerialTransfer myTransfer;

void setup()
{
 Serial.begin(115200);
 Serial1.begin(115200);
 myTransfer.begin(Serial1);
}

void loop()
{
 float myFloat = 100.5;
 
 myTransfer.txBuff[0] = 'h';
 myTransfer.txBuff[1] = 'i';
 myTransfer.txBuff[2] = '\n';
 myTransfer.txFloat(myFloat, 3); //insert the float "myFloat" at index 3 since "hi\n" already takes up indicies 0-2
 
 myTransfer.sendData(7); //3 bytes for "hi\n" plus 4 bytes for the float "myFloat"
 delay(100);
}




RX Arduino:


#include "SerialTransfer.h"

SerialTransfer myTransfer;

void setup()
{
 Serial.begin(115200);
 Serial1.begin(115200);
 myTransfer.begin(Serial1);
}

void loop()
{
 float myFloat;
 
 if(myTransfer.available())
 {
   /////////////////////////////////////////////////////////////// Handle Entire Packet
   Serial.println("New Data");
   for(byte i = 0; i < myTransfer.bytesRead; i++)
     Serial.write(myTransfer.rxBuff[i]);
   Serial.println();

/////////////////////////////////////////////////////////////// Parse Out Float From Packet
   myTransfer.rxFloat(myFloat, 3);
   Serial.print("Received float: "); Serial.println(myFloat);
   Serial.println();
 }
 else if(myTransfer.status < 0)
 {
   Serial.print("ERROR: ");
   Serial.println(myTransfer.status);
 }
}

Thanks for sharing that! Although the sender in my use case may not be using the Arduino framework, I will take a look at that library.

lekok:
What if I want to allow varied payload size? I thinking if the sender sends something like: ><123>
Where '<' and '>' are start/end markers respectively.

If you have start and end markers the receiving code does not need to worry about the size. It just keeps collecting bytes until the end-marker arrives. Just make sure you have allocated enough space to receive the longest message.

...R

Robin2:
If you have start and end markers the receiving code does not need to worry about the size. It just keeps collecting bytes until the end-marker arrives. Just make sure you have allocated enough space to receive the longest message.

...R

Sorry, I may be misinterpreting your answer. But based on what you said, wouldn't my above example: ><123> be treated as two separated batch? To clarify, the example should be treated as one batch. But because '<' and '>' can appear in the payload, just checking for end marker might terminate the reception of the batch early.

lekok:
To clarify, the example should be treated as one batch. But because '<' and '>' can appear in the payload,

Then the '>' can't be used as an end-marker.

Post a couple of examples of complete messages.

Can you change the format of the data being transmitted? Maybe you could use a different end-marker that never appears in the message.

...R

Robin2:
Then the '>' can't be used as an end-marker.

Post a couple of examples of complete messages.

Can you change the format of the data being transmitted? Maybe you could use a different end-marker that never appears in the message.

...R

The idea is that data can appear in the different format and vary in sizes. So for example, it can arrive as 0x6244 which represents 25156 in 16 bits unsigned integer. In this case the 0x62 would have been recognized as the the end marker. But in reality, it is a data byte.

The messages format is entirely up to the sender and can change because the Arduino together with the radio is essentially acting as a transmitter to transmit whatever data it is told. Hence, have to account for all 256 values of a byte.

Do you think this is wishful thinking that I can process this sort of specification with just using markers? Seems like its complex enough to warrant some sort of protocol between the sender and the Arduino.

lekok:
The messages format is entirely up to the sender and can change because the Arduino together with the radio is essentially acting as a transmitter to transmit whatever data it is told. Hence, have to account for all 256 values of a byte.

I think it's time you provided a full description of the project.

When I read about a requirement as wide open as this I get suspicious that someone is trying to do something illegal or immoral.

Assuming this is a legitimate project if we know what you are actually trying achieve someone might be able to suggest a simple solution.

...R