Problems clearing input buffer with serial communcation

[austinallchurch]

Hello all, I am trying to write a straightforward program where the tx and rx lines are connected on my Arduino and I am trying to send the same string iteratively. I have read into using memset and setting the first index of the input buffer with the null terminator, however, it appears that the data within the input buffer is not being properly cleared. If someone would be kind enough to look at my code and the output it would be greatly appreciated!

char version[] = "Hello world";
const unsigned int numChars = 10;

void setup() {
  Serial.begin(9600);

  while(!Serial){
    ;
  }
  
  delay(2000);

}

void loop() {
  char receivedChars[numChars];
  memset(receivedChars, '\0', numChars);

  Serial.write(version);
  delay(10);
  int idx = 0;
  while(Serial.available() > 0 && idx < numChars+1){
      char tempByte = Serial.read();
      receivedChars[idx] = tempByte;
      idx++;

  }
  Serial.println(receivedChars);
  delay(3000);
}

Thanks in advance for the help.

Cheers

[jremington]

To clear the serial input buffer, you can use something like this:

 char tempByte; 
 while(Serial.available() > 0 ) tempByte = Serial.read();

This is guaranteed to write past the end of the allocated buffer, and possible crash the Arduino. Remove the +1.

&& idx < numChars+1)

This can store 9 ASCII characters, leaving room for the required zero terminator byte:

  char receivedChars[numChars];

[austinallchurch]

Hi @Delta_G, thanks for the response! I am trying to write what is in version ("Hello World") so that it goes through the rx line and is received on the tx line. I do believe write is the correct function, could you provide more explanation as to what you mean by it should be print and not write?

[austinallchurch]

Ok that makes sense, I have resolved my array indexing issue. The end goal of this code is to send commands to a GNSS received, would print still be the correct command in this case?

[austinallchurch]

I think most of the initial problems that you very nice people have pointed out to me have been resolved (hopefully). I have chose to keep the write function over the print as I will be sending strings of commands to a GNSS received and write seems to do a better job handling strings. Furthermore, here is my updated code, however there still seems to be some random garbage within the serial that I cannot quite pin point>

char version[14] = "#Hello world*";

const byte numChars = 200;
char receivedChars[numChars];
boolean newComData = false;

void setup() {
  Serial.begin(9600);

  while(!Serial){
    ;
  }
  delay(2000);
}

void loop() {
  //Sending the message
  Serial.println("Sending the following message, expected return without start and end markers:");
  Serial.println(version);
  Serial.println();
  Serial.write(version);

  //Checking to see if data has arrived, if so, parsing it
  recvWithStartEndMarkers();

  //Checking the boolean return from recvWithStartEndMarkers
  if (newComData == true){
    Serial.println("The following message has arrived:");
    Serial.println(receivedChars);
    Serial.println();
    newComData == false;
  }
  else{
    Serial.println("No new data");
  }

  delay(5000);
}

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '#';
  char endMarker = '*';
  char rc;
  while (Serial.available() > 0 && newComData == false) {
    rc = Serial.read();
    if (recvInProgress == true) { // this one is the second
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else { //this is the last, notice the while loop loser on the newComData=true
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newComData = true;
      }
    }
    else if (rc == startMarker) { //this one gets executed first, recvInProgress = true, and then the while do the loop, resulting the if(recvInProgress -- true) started
      recvInProgress = true;
    }
  }
}

The output is as follows:

Most aspects of the code seems to be working properly, however, the extra '#Hello world*' in my return statement 'The following message has arrived' should not be there. Im very happy that the receivedChars buffer is now containing the proper string, but I am concerned about the extra '#Hello World*' as I fear that the communication between arduino and GNSS card will be skewed.

[codingapacalyspe]

In your code snippet, you are trying to read data from the serial buffer and store it into receivedChars. However, there seems to be a logical error in your while loop condition. You are checking for Serial.available() to be greater than zero and also checking if idx is less than numChars + 1. This could lead to an off-by-one error because arrays in C++ are zero-indexed. Therefore, you should check against numChars, not numChars + 1.

Here's the corrected part of your code:

while(Serial.available() >  0 && idx < numChars){
    char tempByte = Serial.read();
    receivedChars[idx] = tempByte;
    idx++;
}

Additionally, it's good practice to flush the input buffer before you start reading to ensure that old data doesn't interfere with your new reads. You can use Serial.flush() to achieve this:

void loop() {
  char receivedChars[numChars];
  memset(receivedChars, '\0', numChars);

  Serial.write(version);
  delay(10);
  Serial.flush(); // Flush the input buffer

  int idx =  0;
  while(Serial.available() >  0 && idx < numChars){
      char tempByte = Serial.read();
      receivedChars[idx] = tempByte;
      idx++;
  }
  Serial.println(receivedChars);
  delay(3000);
}

Remember to call Serial.flush() after writing to the serial port to ensure that all data is sent before you attempt to read it back. This should help you clear the input buffer effectively before starting a new read cycle.

char version[] = "Hello world";
const unsigned int numChars = sizeof(version) -  1; // Adjusted to get the length of the string

void setup() {
  Serial.begin(9600);

  while (!Serial) {
    ; // Wait for serial port to connect. Needed for native USB port only
  }

  delay(2000);
}

void loop() {
  char receivedChars[numChars +  1]; // Added one extra space for null terminator
  memset(receivedChars, '\0', sizeof(receivedChars)); // Use sizeof to set the correct size

  Serial.print(version); // Use print instead of write to avoid sending binary data
  delay(10);
  Serial.flush(); // Clear the input buffer before reading

  int idx =  0;
  while (Serial.available() >  0 && idx < numChars) {
    char tempByte = Serial.read();
    receivedChars[idx] = tempByte;
    idx++;
  }

  Serial.println(receivedChars); // Print the received string
  delay(3000);
}

[austinallchurch]

Thanks for the reply @codingapacalyspe, if you go down in the thread I resolve must of the issues that you are mentioning! Thank you so much for taking the time to review my code though

[codingapacalyspe]

The statement that Serial.flush() blocks to clear the output buffer and has nothing to do with the input buffer is incorrect. The Serial.flush() method in Arduino programming is intended to ensure that all outgoing serial data has been transmitted and that the output buffer is empty before proceeding with further operations. This can be useful to ensure synchronization between the sender and receiver, especially in cases where timing is critical.

In contrast, Serial.flush() does not interact with the input buffer. To clear the input buffer, you would typically read from the buffer until it is empty, or use a method like Serial.clear() if available. However, the standard Arduino Serial library does not provide a direct method to flush the input buffer; it only provides methods to read from the buffer or check how many bytes are available (Serial.available()).

For PySerial, a Python library that provides serial communication capabilities, there is a distinction between flush(), reset_input_buffer(), and reset_output_buffer(). The flush() method sends all data in the output buffer to the peer, whereas reset_output_buffer() discards the data in the output buffer without sending it [0]. There is no direct equivalent to Serial.flush() for input buffers in PySerial; instead, you might use reset_input_buffer() to clear the input buffer [0].

Here's an example of using Serial.flush() in Arduino:

Serial.write("Hello"); // Write data to the serial port
Serial.flush();       // Wait for the transmission to complete
// Now the output buffer is cleared and we can proceed with more writes or reads

And here's a hypothetical example of using reset_input_buffer() in PySerial (note that this method does not exist in the PySerial documentation, but is used for illustrative purposes):

import serial

ser = serial.Serial('/dev/ttyS0')  # Open the serial port
ser.write(b'Hello')                # Write data to the serial port
ser.reset_input_buffer()           # Clear the input buffer
# Now the input buffer is cleared and we can proceed with more writes or reads

Remember, when using PySerial, it's important to understand the differences in behavior and functionality compared to Arduino's Serial library, as they are two different implementations of serial communication handling.

[austinallchurch]

Thanks for the response @Delta_G. I hope I didn't offend you in my last message regarding the print vs write dilemma, my use case will be a fixed command i.e. (log versiona) which will be sent to the GNSS card. It will also have a known size. From my ground level understanding, the write function will take each character and write them into ascii form where the serial.read can decode the characters (If I am mistaken my apologies).

Additionally, I am under the impression that receivedChars should only hold "Hello World" as within my recvWithStartEndMarkers function, it should remove the start and end characters and store the string within this input buffer. It isn't called anywhere else so I would be slightly confused if the line

Serial.println(receivedChars);

Was causing the issue

[austinallchurch]

Thanks for the clarification around the flush within input and output buffers! I do not believe that a Serial.flush() would fix my issue as I am reading the serial data until the statement

while (Serial.available() > 0)

becomes false, inturn, flushing the output buffer (Im pretty sure but correct me if I'm wrong). Additionally, the only time the string "#Hello World*" is issued is in the input buffer. Maybe this is my problem and that somehow the input buffer is printing to my serial monitor.

Lastly, does Serial.write() print the string to the serial monitor?

[codingapacalyspe]

Yes, Serial.write() prints the string to the serial monitor in the form of raw binary data. This means that if you use Serial.write("Hello World*"), the exact sequence of bytes representing the string "Hello World*" will be sent over the serial connection, and it will appear on the serial monitor as such. The serial monitor does not interpret the incoming data as text unless it is specifically formatted in ASCII or another human-readable encoding.

On the other hand, Serial.print() takes care of converting the data into human-readable ASCII text before sending it over the serial connection. Therefore, if you use Serial.print("Hello World*"), the string "Hello World*" will be displayed on the serial monitor as the literal text "Hello World*" .

Regarding your concern about the input buffer, if the string "#Hello World*" is being issued only in the input buffer and you suspect it is causing issues, you may want to ensure that your reading logic is correctly identifying the end of the message and that you are properly handling any leftover data. One common practice is to use a delimiter or specific termination character to signal the end of a message, and then process the message accordingly. If you're experiencing unexpected output on the serial monitor, it might be due to unhandled or improperly processed input data.

Here is an example of how you might handle incoming serial data with a specific delimiter:

String inputString = "";
char incomingByte;

void setup() {
  Serial.begin(9600);
}

void loop() {
  if (Serial.available() >  0) {
    incomingByte = Serial.read();
    if (incomingByte != '#') {
      inputString += incomingByte;
    } else {
      // Process the received message
      Serial.println(inputString);
      inputString = "";
    }
  }
}

In this example, the program waits for the '#' character to indicate the end of a message before processing and printing the message to the serial monitor. This helps ensure that you are not accidentally printing incomplete messages or leftover data from previous communications.

[austinallchurch]

Thank you very much for the clarification! Removing serial.println(version) gave me the expected return! I was unware that serial.write() displayed on the serial monitor, thanks a ton.

[david_2018]

It actually does not make any difference whether you use print() or write() when using a null-terminated char array. There is an overload of write() specifically for char arrays, found in Print.h. As far as the receiving device is concerned, the exact same data is being sent over serial, and it is indistinguishable which function was used to send it.

    size_t write(const char *str) {
      if (str == NULL) return 0;
      return write((const uint8_t *)str, strlen(str));
    }