Serial.available & read doing two different things on similar programs?

Robin2's Serial Input Basics was good intro on communicating with the Arduino, but one thing that I'm still confused about is the Serial.available() and read() functions. Like the Serial Reference page and Robin2 stated, Serial.flush doesn't actually clear any buffer.

From the Reference page, Serial.available "gets the number of bytes available for reading from the serial port. This is the data that's already arrived and stored in the serial buffer, which holds 64 bytes". This should mean that Serial.available returns a int(I assume) But if a, while (Serial.available() > 0), was by itself (with no Serial.read), this code would be an infinite loop since it'll always stay true.

Serial.read reads incoming serial data. Is it technically from the data stored in the serial buffer, or was the buffer only used in determining the number of bytes in the stream? Nowhere does it state that it removes data from either the serial buffer or incoming data? There have been other questions on how to clear the serial buffer, and they all say to just use Serial.read. Then there is this post where " throwing away serial data without any regard for what is there is generally considered stupid".

So my understanding is since it seems when Serial.read executes and reads the data, it removes it from either the stream/buffer which results in a decrease in the value of Serial.available. This should mean that when used in conjunction. Serial read and available should read all the data from the buffer? If executed properly, this should mean that there wouldn't be a need to throw away serial data without regard of its content and needing a function to just wipe data irregardless shouldn't be probably, or would be a result of a mismatch in the program itself?

Why I bring this up is because I wrote two programs to read data from the Serial Com and they complete the same function, but one continuously executes a blink function within the while loop constantly, whereas the other only executes it following an input. I can't seem to figure out why these two are behaving differently.

The only code is basically the same between the two. Other than the loop and receiving function. The first program requires a delay after receiving in order to even execute the rest of the code. The only other difference I'm noticing is that in one program is that one is set with Start and End markers, while the other has none.

This code continuously blinks my led after sending data

void loop()
{
    recvBytesWithEndMarkers();//get Data
    //delay(500);
    S1();
    S2();
    S3();
    S4();
    S5();
    S6();
    S7();
    S8();
    //Serial.flush();//Clear Input buffer
    blink();
    //resetbits();
}
void recvBytesWithEndMarkers() {
    boolean recvInProgress = false;
    int ndx = 0;
    String RXString;

    while (Serial.available() > 0/* && newData == false*/)
    {
        RXString = Serial.readStringUntil('\n');

        if (recvInProgress == true)
        {
            if (RXString != EMS) // if RXString[] != endmarker[]
            {
                receivedString[ndx] = RXString;
                copyarray[ndx] = receivedString[ndx];
                ndx++;
                if (ndx >= numBytes)
                {
                    ndx = 7;
                }
            }
            else
            {
                receivedString[7] == EMS; // terminate the string
                recvInProgress = false;
                ndx = 0;
                //newData = true;
            }
        }
        else if (RXString == SMS)
        {
            recvInProgress = true;
        }
    }
}

This code only blinks after you send the data, and doesn't blink until you send it new data

void loop()
{
    getstring();
    S1c();
    S2c();
    S3c();
    S4c();
    S5c();
    S6c();
    S7c();
    S8c();
    blink();
    delay(1000);
}
void getstring()
{
    int ndx = 0;
    String rs;
    while (Serial.available() > 0)
    {
        rs = Serial.readStringUntil('\n');
        rarray[ndx] = rs;
        carray[ndx] = rarray[ndx];
        ndx++;
        if (ndx > 7)
            break;
    }
}

The SX functions are copy and pasted and the same between the programs and are just String to Int conversions of an element in the array. Since the code executes, and my LED's do blink, I know I'm not in any sort of infinite loop while reading, so only the Ardino's loop is forcing the LED's to constantly blink.

What I'm really puzzled about and want to understand is, how come my second code blinks only when I send data? I would assume that since the blink codes are within the loop, they'll be called continuously as the Arduino UNO is running, and since I didn't reset the values of any elements, they'll blink to the previous input until they're told to blink to sometime else. In the first code if I uncomment the resetbits function, it'll only run the inputted commands once, since after it is ran, they're reinitialized to 0, but in my second program, I did no such thing, and it only ran once.

If you want to understand what's going on, you need to read up on ring or FIFO buffers and C-strings. A C-string is an array of characters terminated by a NULL (\0). In the case of the Serial class for Arduino, the read buffer is 64 bytes which allows it to receive up to 63 characters before overwriting (one position must be reserved for \0). When you call the Serial.available() function, it invokes the strlen() library function that return the number of characters in the array until (but not including) \0.

When you push and pop from a ring buffer, a value is either inserted at one end or removed from the other. The NULL terminator is adjusted accordingly. As you successive read (pop) data from the buffer, the NULL terminator will eventually end up in buffer position zero at which time a call to strlen will result in zero and if(Serial.available()) will fail.

    receivedString[7] == EMS; // terminate the string

This line does nothing. If it was supposed to do something it is written incorrectly.

Perhaps you should set your warning level to ALL and check your code for mistakes. That sometimes turns up mistakes that are not strictly illegal but are sometimes fatal.

This may be separate from your particular issue - but: Serial is slow (9600 baud is just over 1 millisecond per character). The only reason either of your programs happen to work most of the time is that the data is coming in during the delay(). If data happened to be in the process of coming in when your function read from the serial port, you'd only get part of the data.

You don't seem to have absorbed Robin2's serial input basics. Usually what you do is - as data comes in the serial port, you stuff it into another buffer until you hit a character that signals the end of the input (I use one at the start and end of the input), then pass that buffer off to a function that processes it. This of course typically also requires using millis() rather than delay() for timing.

In good old days of DOS/C/XT/UART Programming, we first executed a line of code to check that a character has really arrived at the receiver section of the serial port and then we executed another line of code to read the arrived data byte from the receiver section. This was to avoid the wastage of time as it was useless to make an attempt to read a data byte from the receiver section when there is no character that has arrived.

In Arduino Serial Platform, the Serial.available() and the Serial.read() function exactly do the above two tasks.

In addition, the Arduino has incorporated an unseen 64-byte wide FIFO buffer to accommodate charcaters as they arrive (provided the characters are not read when they have arrived). The buffer, in fact, can store 63 characters as the last byte is a null-byte. As the FIFO buffer is not directly accessible by the user, the program must execute the Serial.read() instruction to bring the content of the FIFO buffer into the user defined variable.

The following program shows that the buffer keeps accommodating 5 unread charcaters (say, a b c d e) as they are entered one at a time from the InputBox of the Serial Monitor (No line ending option should be active); at the end of the reception of 5 characters, the Serial Monitor shows abcde in the OutputBox of the Serial Monitor.

char myData[20] = "";

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  if (Serial.available() > 0)
  {
    byte n = Serial.available();
    if ( n == 5)
    {
      for (int i = 0; i < n; i++)
      {
        myData[i] = Serial.read();
      }
      Serial.println(myData);
    }
  }
}

jim-yang: So my understanding is since it seems when Serial.read executes and reads the data, it removes it from either the stream/buffer which results in a decrease in the value of Serial.available. This should mean that when used in conjunction. Serial read and available should read all the data from the buffer?

That sounds correct and is what my Serial Input Basics code does. Serial.read() removes a byte from the Serial Input Buffer.

If executed properly, this should mean that there wouldn't be a need to throw away serial data without regard of its content and needing a function to just wipe data irregardless shouldn't be probably, or would be a result of a mismatch in the program itself?

This seems to be raising a different issue and I don't understand what was in your mind when you wrote it. I can think of good reasons why one would wish to discard data - for example if you could not be sure that it is valid.

...R

   while (Serial.available() > 0)
   {
       rs = Serial.readStringUntil('\n');

Serial.readStringUntil() does not work the same as Serial.read(), and will set there and read any number of characters are received, until it finally sees the '\n'; that's the whole point of the "Until" part of the name.

Serial.read reads incoming serial data. Is it technically from the data stored in the serial buffer

Yes, it's from the serial buffer.

Nowhere does it state that it removes data from either the serial buffer or incoming data?

That's pretty much "standard behavior" for "read" functions of "normal" ("buffered", or "stream") devices. Reading data removes it from the queue of data that is available to be read. I guess Arduino has an abnormally high number of non-buffered devices that can be read (eeprom_read(), pgm_read_byte(), digitalRead()), analogRead())

DKWatson:
If you want to understand what’s going on, you need to read up on ring or FIFO buffers and C-strings. A C-string is an array of characters terminated by a NULL (\0). In the case of the Serial class for Arduino, the read buffer is 64 bytes which allows it to receive up to 63 characters before overwriting (one position must be reserved for \0). When you call the Serial.available() function, it invokes the strlen() library function that return the number of characters in the array until (but not including) \0.

When you push and pop from a ring buffer, a value is either inserted at one end or removed from the other. The NULL terminator is adjusted accordingly. As you successive read (pop) data from the buffer, the NULL terminator will eventually end up in buffer position zero at which time a call to strlen will result in zero and if(Serial.available()) will fail.

I know about ring and FIFO buffers and have read about c strings. Not sure what your point about the null character is since they null character isn’t transmitted. Yes I do understand it’s there, and I have much less than a total of 63 characters. I don’t think I’m doing a ring buffer, pretty sure it’s just a normal FIFO buffer I’m dealing with?

This line does nothing. If it was supposed to do something it is written incorrectly.

Perhaps you should set your warning level to ALL and check your code for mistakes. That sometimes turns up mistakes that are not strictly illegal but are sometimes fatal.

I just wanted to make sure the data I had was a equal match at the end, kind of like a fail safe? Since it’s the end marker, the ending should be the marker? If somehow some other element was written into the last slot, it would mess up the endmarker, so by setting the endmarker up, I would make sure the marker wasn’t overridden.

DrAzzy:
This may be separate from your particular issue - but: Serial is slow (9600 baud is just over 1 millisecond per character). The only reason either of your programs happen to work most of the time is that the data is coming in during the delay(). If data happened to be in the process of coming in when your function read from the serial port, you’d only get part of the data.

You don’t seem to have absorbed Robin2’s serial input basics. Usually what you do is - as data comes in the serial port, you stuff it into another buffer until you hit a character that signals the end of the input (I use one at the start and end of the input), then pass that buffer off to a function that processes it. This of course typically also requires using millis() rather than delay() for timing.

I know it’s slow, and don’t understand your comment. As to the only reason it works is because the data came in the delay, I know that? That’s kind of why I added the delay I assume you understand that? I tried playing with some of the other bauds and some of the worked, others I had issues, 9600 and 19200 worked so I’ve been sticking with those. I played around with millis(), but delay pauses the whole program, so I can worry about reading and writing one at a time.

GolamMostafa:
In good old days of DOS/C/XT/UART Programming, we first executed a line of code to check that a character has really arrived at the receiver section of the serial port and then we executed another line of code to read the arrived data byte from the receiver section. This was to avoid the wastage of time as it was useless to make an attempt to read a data byte from the receiver section when there is no character that has arrived.

In Arduino Serial Platform, the Serial.available() and the Serial.read() function exactly do the above two tasks.

In addition, the Arduino has incorporated an unseen 64-byte wide FIFO buffer to accommodate charcaters as they arrive (provided the characters are not read when they have arrived). The buffer, in fact, can store 63 characters as the last byte is a null-byte. As the FIFO buffer is not directly accessible by the user, the program must execute the Serial.read() instruction to bring the content of the FIFO buffer into the user defined variable.

The following program shows that the buffer keeps accommodating 5 unread charcaters (say, a b c d e) as they are entered one at a time from the InputBox of the Serial Monitor (No line ending option should be active); at the end of the reception of 5 characters, the Serial Monitor shows abcde in the OutputBox of the Serial Monitor.

char myData[20] = "";

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  if (Serial.available() > 0)
  {
    byte n = Serial.available();
    if ( n == 5)
    {
      for (int i = 0; i < n; i++)
      {
        myData[i] = Serial.read();
      }
      Serial.println(myData);
    }
  }
}

GolamMostafa:
In good old days of DOS/C/XT/UART Programming, we first executed a line of code to check that a character has really arrived at the receiver section of the serial port and then we executed another line of code to read the arrived data byte from the receiver section. This was to avoid the wastage of time as it was useless to make an attempt to read a data byte from the receiver section when there is no character that has arrived.

In Arduino Serial Platform, the Serial.available() and the Serial.read() function exactly do the above two tasks.

In addition, the Arduino has incorporated an unseen 64-byte wide FIFO buffer to accommodate charcaters as they arrive (provided the characters are not read when they have arrived). The buffer, in fact, can store 63 characters as the last byte is a null-byte. As the FIFO buffer is not directly accessible by the user, the program must execute the Serial.read() instruction to bring the content of the FIFO buffer into the user defined variable.

The following program shows that the buffer keeps accommodating 5 unread charcaters (say, a b c d e) as they are entered one at a time from the InputBox of the Serial Monitor (No line ending option should be active); at the end of the reception of 5 characters, the Serial Monitor shows abcde in the OutputBox of the Serial Monitor.

char myData[20] = "";

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  if (Serial.available() > 0)
  {
    byte n = Serial.available();
    if ( n == 5)
    {
      for (int i = 0; i < n; i++)
      {
        myData[i] = Serial.read();
      }
      Serial.println(myData);
    }
  }
}

Isn’t what you’re describing just part of debugging today? So if I understand correctly in the code you wrote, it doesn’t read data until 5 bytes have been entered? I have thought about a solution like this with fixed digits, but didn’t think about using byte n = Serial.available() as a declaration to count the number of bytes to be read

Robin2:
That sounds correct and is what my Serial Input Basics code does. Serial.read() removes a byte from the Serial Input Buffer.

This seems to be raising a different issue and I don’t understand what was in your mind when you wrote it. I can think of good reasons why one would wish to discard data - for example if you could not be sure that it is valid.

…R

As you mentioned, I think there are good reasons why you would wish to clear data, but I was thinking that it’s ill advised to get into the situation in which you weren’t sure what data was valid. Another user was making a comment about why Serial.flush couldn’t just discard the data and how it’d be so nice, and another user saying that would be a terrible option and it’s good that Serial.flush does what it does. I think both comments make good points.

I would think that in general practice you’d plan your code to not be in that sort of situation. For my case, I want to send a 8 element string array, with 1 start and 1 end element, and 6 data elements. What I meant in my comment was, It’d be most efficient, to read the whole array, save each element, and then use them in other functions. If I send 8 elements and plan to read 8 elements, there should technically be no more elements to be read and therefore no need to discard any data. The least efficient method in my opinion would be to send 8 elements, read 1 element, discard any data leftover, and repeat for the next 7 elements.

westfw:

   while (Serial.available() > 0)

{
      rs = Serial.readStringUntil(’\n’);




Serial.readStringUntil() does not work the same as Serial.read(), and will set there and read any number of characters are received, until it finally sees the '\n'; that's the whole point of the "Until" part of the name.Yes, it's from the serial buffer.That's pretty much "standard behavior" for "read" functions of "normal" ("buffered", or "stream") devices. Reading data removes it from the queue of data that is available to be read.
I guess Arduino has an abnormally high number of non-buffered devices that can be read (eeprom_read(), pgm_read_byte(), digitalRead()), analogRead())

I understand that Serial.readStringUntil() and Serial.read() are different, I have planned for such case since in my windows form, I use WriteLine method, in which the data rights the bytes as well as a New Line character ‘\n’. I didn’t post that part of the code, my apologies, since I didn’t think it was relevant.

I would have thought that the Serial.read would just read the stored data in the register/buffer (I know that the Arduino uses registers, but not much else), read it and then move on with the program. I would have thought that a command that would remove the data from said register/buffer when used would have been named something like Serial.take or use.

I wasn't exactly aware that Serial.read removes data from the buffer so I was wondering if there was anything else that might be occurring that I'm not aware of. One thing I am not really sure of was, if the Serial.read and Serial.available pause or hold the program for any reason? As the only difference between the two codes were the way the data was being handled, I'm unaware of what is making one program continuously executing my blink command, where as the other or executes the command after data is received.

Since the commands are within the loop of the Arduino itself, my assumption was that blink function should be executed constantly after the first iteration, since even if no data is currently in the input buffer, the previous iteration are already loaded into the blink function. This means that the other functions would still be continuously running. If I uncomment my reset function, it sets all the values to be 0 at the end of the loop. Therefore blink only executes once. In my second program, I didn't not implement any sort of reset function, but the blink function only executes after it receives data.

What my assumption was, when I was posting this was: In my first program, it reads all available data, finishes and then continues with the loop. This would be why it constantly executes the blink function. In the second program, the loop gets stuck during the Serial.read and available area. This means that until new data is entered, the loop doesn't continue onto the blink function. I'm starting to think that this hypothesis is wrong.

DKWatson: If you want to understand what's going on, you need to read up on ring or FIFO buffers and C-strings. A C-string is an array of characters terminated by a NULL (\0). In the case of the Serial class for Arduino, the read buffer is 64 bytes which allows it to receive up to 63 characters before overwriting (one position must be reserved for \0). When you call the Serial.available() function, it invokes the strlen() library function that return the number of characters in the array until (but not including) \0.

When you push and pop from a ring buffer, a value is either inserted at one end or removed from the other. The NULL terminator is adjusted accordingly. As you successive read (pop) data from the buffer, the NULL terminator will eventually end up in buffer position zero at which time a call to strlen will result in zero and if(Serial.available()) will fail.

I read up on this and looked at the source code.

The read buffer will in fact hold 64 characters.

There is no placement or need for any '\0' in the buffer.

Serial.available() does not invoke strlen(), it uses standard ring/circular buffer algos. Understand this to get the idea of the circular buffer, viz:

int HardwareSerial::available(void)
{
  return ((unsigned int)(SERIAL_RX_BUFFER_SIZE + _rx_buffer_head - _rx_buffer_tail)) % SERIAL_RX_BUFFER_SIZE;
}

How would strlen() even work in a circular buffer?

a7

jim-yang: but I was thinking that it's ill advised to get into the situation in which you weren't sure what data was valid.

I agree. That's the purpose of the 3rd example in Serial Input Basics

Note, however, that it is designed to discard characters until it detects the start-marker. And then it only reports a valid message when it detects the end-marker.

For additional security you could include a checksum in the message - but that is not really a "serial" issue.

...R