Arduino stops receiving serial characters into the buffer after some are received (less than 5 in the buffer)

Hello!
I have a weird issue, and after chatGPT had no idea, I think it may be a bug but before I submit an issue, let's see if someone can help me out.

I have some code that does serial communication. My issue is that if I enter the data in a specific way it completely stops receiving data over serial.

This code expects either a INITMARKER ('g') which would make it start running normally, or a ONEMSGMARKER ('o') which just gives one message to the Arduino without ack or anything like that.

I had an issue of the sending system sometimes sending half the characters, getting interrupted, then sending the other half. So I made my code wait until there are enough characters in the serial buffer before executing. Unfortunately, for whatever reason if I send in a particular way, it never fills the buffer with them.

This works well:
o(oo) send
o send (oo) send

This does not work:
o( send oo) send
o(o send o) send
o(oo send ) send
o send ( send o send o send ) send

In all cases that do not work, whatever characters are sent the first time around are seen in the serial buffer (Serial.available() shows them), but then sending more characters does not increment Serial.available() and the code never executes.

Here is the relevant part of the communication function:

if (Serial.peek() == ONEMSGMARKER){
    if (Serial.available() >= 5){
        Serial.read(); //discard the peeked character
        Serial.println("About to go into the loop");
        while (Serial.available()){
            recvChar = Serial.read();
            if (bitRead(recvChar,POS_SERIAL_RES)){ //if the resrved bit is set, this is not part of the data
                Serial.println(F("!Did not expect an ACK/NCK char. Clearing buffer!"));
                while (Serial.available()){
                    Serial.read();
                }
                return;
            }
            else if (recvChar == STARTMARKER){
                index = 1;
            }
            else if (index) {
                if (recvChar != ENDMARKER) {
                    if (index > SERIAL_INDEX_MAX){
                        Serial.println(F("!Message too long!"));
                    }
                    message += recvChar << ((index-1)*8);
                    index++;
                }
                else { // end marker
                    index = 0;
                    serialFromPi = message;
                    break; //the recvChar loop
                }
            }
        }
    }
    else{
        Serial.print(F("In buffer:"));
        Serial.println(Serial.available());
    }
}

Here is the entire function. void loop() just calls this and other non serial related functions over and over.

uint8_t talkToPi() {
    //state machine variables
    static uint8_t state = SERIAL_STARTING;
    //timing variables
    uint32_t ms = millisec();
    static uint32_t prevReplyMS = 0, prevSendMS = 0;
    //recieve variables
    static uint8_t index = 0;
    char recvChar;
    static uint16_t message;
    //recieve variables are defined for the entire function above
    //send variables
    static uint16_t prevMessageToPi = 0x4747; //GG
    static uint8_t prevMessageAckByPi, uniqueMessagesMissed;

    switch (state){ 
        case SERIAL_STARTING:
            if(Serial){
                state = SERIAL_INITIALIZING;
                break;
            }
            Serial.begin(0); //No baudrate for USB serial
            break;
    
        case SERIAL_INITIALIZING:
            message = 0; // this is here in case ONEMSGMARKER is sent and the data has the reserved bit set. The function will return and discard the data 
            if (Serial.peek() == ONEMSGMARKER){
                if (Serial.available() >= 5){
                    Serial.read(); //discard the peeked character
                    Serial.println("About to go into the loop");
                    while (Serial.available()){
                        recvChar = Serial.read();
                        if (bitRead(recvChar,POS_SERIAL_RES)){ //if the resrved bit is set, this is not part of the data
                            Serial.println(F("!Did not expect an ACK/NCK char. Clearing buffer!"));
                            while (Serial.available()){
                                Serial.read();
                            }
                            return;
                        }
                        else if (recvChar == STARTMARKER){
                            index = 1;
                        }
                        else if (index) {
                            if (recvChar != ENDMARKER) {
                                if (index > SERIAL_INDEX_MAX){
                                    Serial.println(F("!Message too long!"));
                                }
                                message += recvChar << ((index-1)*8);
                                index++;
                            }
                            else { // end marker
                                index = 0;
                                serialFromPi = message;
                                break; //the recvChar loop
                            }
                        }
                    }
                }
                else{
                    Serial.print(F("In buffer:"));
                    Serial.println(Serial.available());
                }
            }
            
            else if (Serial.peek() == INITMARKER){
                Serial.read(); //discard the peeked character
                state = SERIAL_RUNNING;
            }

            else if (Serial.peek() != -1 && Serial.availableForWrite() > 32){ // If a bad initialize character(s) is recieved, repeat evrrything back with an error message
                Serial.print(F("!Unkown start char from Pi:")); //documentation: what's F
                while (Serial.available() > 0 && Serial.availableForWrite() > 1){
                    recvChar = Serial.read();
                    Serial.print(recvChar);
                }
                Serial.println("!");
            }   

            // Send start character if pi has initialized
            if (state == SERIAL_RUNNING){
                Serial.write(INITMARKER); //should block if the buffer is full. If it doesnt, Pi may never be initialized
                prevReplyMS = ms;
            }
            break;
        
        case SERIAL_RUNNING:
            //Serial.println("here!");

            // Recieve a packet
            if (Serial.available() >= 4){
                //Serial.
                prevReplyMS = ms;
                while (Serial.available()){     
                    recvChar = Serial.read(); 
                    
                    //the recieved char may either be ack/nck, start/end marker, or data:
                    if (bitRead(recvChar,POS_SERIAL_RES)){ //if the resrved bit is set, this is not part of the data
                        if (recvChar == ACKMARKER || recvChar == NAKMARKER){
                            prevMessageAckByPi = recvChar;
                        }
                        else{
                            Serial.println(F("!Bad ACK/NCK char!"));
                        }
                    }
                    else if (recvChar == STARTMARKER){
                        index = 1;
                    }
                    else if (index) {
                        if (recvChar != ENDMARKER) {
                            if (index > SERIAL_INDEX_MAX){
                                Serial.println(F("!Message too long!"));
                            }
                            message += recvChar << ((index-1)*8);
                            index++;
                        }
                        else { // end marker
                            index = 0;

                            //parity check
                            //parity_even_bit() generates the even parity bit. The parity from the message is removed and a new
                            //parity is generated, which is compared with the recieved parity.
                            if (parity_even_bit(message & ~bit(POS_SERIAL_EPARITY)) == bitRead(message,POS_SERIAL_EPARITY)){
                                if (Serial.availableForWrite() > 32){
                                    Serial.write(ACKMARKER); 
                                    serialFromPi = message;
                                    message = 0;
                                }
                            }
                            else {
                                if (Serial.availableForWrite() > 32){
                                    Serial.write(NAKMARKER); //This sends "NAK" (Negative aknowlegde) character, so the pi repeats
                                } 
                            }

                            break; //the recvChar loop
                        }
                    }
                }
            }
            // If nothing is recieved, lets do a timeout check:
            else if (ms - prevReplyMS > 10000){
                //Serial.print(ms);
                //Serial.print(prevReplyMS);
                Serial.println("!Timeout!");
                state = SERIAL_INITIALIZING;
            }

            //--------------------------------------------------------------
            // Sending a packet
            /* TO BE DONE. Here's the old code, which may work well or not:

            serialToPi = bitClear(serialToPi,POS_SERIAL_EPARITY); //remove old parity bit
            serialToPi += parity_even_bit(serialToPi) << 15; //add parity bit to the start
            if (prevMessageAckByPi == ACKMARKER){
                uniqueMessagesMissed = 0;
                if (Serial.availableForWrite() > 32){
                    if (prevMessageToPi != serialToPi || ms - prevSendMS > SEND_NONUNIQUE_MS){
                        prevSendMS = ms;
                        prevMessageToPi = serialToPi;
                        Serial.write(STARTMARKER);
                        Serial.write(serialToPi >> 8); //1111-0000-1010-1100 becomes >>>>->>>>-1111-0000
                        Serial.write(serialToPi); //Only 1010-1100 is accepted from 1111-0000-1010-1100
                        Serial.write(ENDMARKER);
                        Serial.write(0x0A); //new line        
                    }
                }
                else {
                    #ifndef DEBUGMODE
                        return SERIAL_FAIL;
                    #endif;
                }
            }
            else {
                if (Serial.availableForWrite() > 32){
                    prevSendMS = ms;
                    Serial.write(STARTMARKER);
                    Serial.write(prevMessageToPi >> 8); //1111-0000-1010-1100 becomes >>>>->>>>-1111-0000
                    Serial.write(prevMessageToPi); //Only 1010-1100 is accepted from 1111-0000-1010-1100
                    Serial.write(ENDMARKER);
                    Serial.write(0x0A); //new line
                
                    if (prevMessageToPi != serialToPi){
                        uniqueMessagesMissed++;
                        if (uniqueMessagesMissed > 10){
                            #ifndef DEBUGMODE
                                return SERIAL_FAIL;
                            #endif
                        }
                    }
                }
                else {
                #ifndef DEBUGMODE
                    return SERIAL_FAIL;
                #endif
                }
            }
            */
            break;

        default:
            Serial.println(F("!Bad State!"));
    }
}

Your topic was MOVED to its current forum category which is more appropriate than the original as it has nothing to do with Installation and Troubleshooting of the IDE

I promise I do normally check the pinned posts, but this issue has drained all of my mental capacity for today :sweat_smile:

seriously? ... it's hilarious that you tried though.

How chatty is the sender ? How much time (in ms?) does the non serial related stuff takes in the loop? (Assuming everything is connected correctly with a suitable voltage adapter and GND connected)

Everything is connected correctly, its a custom designed board. Serial communication is done over USB (im using the 32U4 from the Leonardo). Sender is not chatty, in my test scripts I just send each letter one at a time on my Rapsberry Pi. The pseudocode is literally just:

send('o')
send('(')
send('oo')
send(')')

I also tried the same over serial monitor, for example when I say o(oo send ) send, there is a gap of maybe half a second between both? I just reset it, send these two lines, and nothing happens. Not a loop or anything

The other functions accessed via the main loop are ones to service neopixel animation and other similar stuff. Nowhere are interrupts fully disabled, only just briefly, and Serial is only touched in this function. There is some register access in a couple of functions, but again, nothing relating to Serial or permenantly disabling interrupts

Haha yes! It didn't quite admit it had no idea, but it started suggesting random things, for example that it said the Arduino was going into the while loop. I told it that it isnt, because there is a print there that never executes when the problem arises. Then it started to repeat the same things we've already talked about and I gave up.

How many neopixels? Most neopixel code disables interrupts while sending data to the neopixels, and there could be some issues with memory.

Also, are you using any line ending when sending the data? A newline or carriage return would cause problems with your code.

I'm a bit unclear about your protocol.
Did I get right that the RPi sends
'g' to activate serial communication mode
and then whenever there is a message the RPi sends "o(" as the start marker and the message follows until you get ')' as an end marker (or message too long or timeout if you don't get the end marker after some idle time).

As you have a start and end marker I would apply what's described in Serial Input Basics and capture a full frame in a large enough buffer (SERIAL_INDEX_MAX+1) and not try to second guess timings, or do things like

              while (Serial.available()) {
                Serial.read();
              }

because you might be clearing too much and the next start marker and why do you do this type of thing

      else if (Serial.peek() != -1 && Serial.availableForWrite() > 32) { // If a bad initialize character(s) is recieved, repeat evrrything back with an error message

if for whatever reason the output buffer cannot hold your message (why do you make it so long juste send "!NACK" or may be something short with an error code "!N42"...) you don't send it.

You should Just keep it a very simple state machine.

  • ignore everything until you get the 'g' then remember the serial communication is active
  • ignore everything you get until you receive the start marker
  • collect everything that's coming in into the buffer until you get the end marker, check for buffer overflow and timeout (in which case ignore the frame and wait again for the next start marker)
  • validate the frame through parity or checksum or whatever CRC you want to implement. if it's fine, handle the frame to a parsing function for action, if it's wrong ignore the frame and possibly notify the sender

--

side note: I noticed the comment

        Serial.print(F("!Unkown start char from Pi:")); //documentation: what's F

F() is a macro telling the compiler to store the text in Flash memory instead of RAM. it's documented at the very end of the PROGMEM section of the documentation.

@david_2018
There are 20 RGB and 60 RGBW neopixels, serviced 30 times a second or less. I dont think its the issue here. I am not using line end or carriage return. The data may look like one of those as it is a 16 bit binary number with each set of bits carrying information about something.

@J-M-L
Thanks for taking the time to analyze the code, its not exactly what my protocol does, it actually follows the Serial input basics, adding initialization, acknowledgment/repeat, and some other things. How the communication starts is first the link is established, then the Arduino waits to receive a 'g' character. when that happens, the Arduino sends 'g' back to acknowledge and goes into normal run mode in which it expects communication to keep going with each message to and from the Pi getting acknowledged or repeated.

in that case, you would send 'g', wait for it back, then you send and receive data in the format of (_ _) where ( and ) are the start and end characters (in the code these are STARTMARKER and ENDMARKER respectively).

so, either you: send g,wait to get g back, then receive date: (xx)(xx)(xx)(xx) which would timeout if you dont send in a while, or: you just send o(xx) and be done with it during debugging.

On the raspberry pi side, I fixed this just just putting it all into one big b'' and that worked. But Id still like to know why the arduino stops receiving into its serial buffer

If it is helpful at all, here are the relevant definitions:

//Communication definitons 
  //Parsing macro (like a function), it extarcts the data you need
  #define PARSE(VAR,POS,MASK)   (VAR>>POS)&MASK
  //Program flow
  #define SERIAL_STARTING       0b0
  #define SERIAL_INITIALIZING   0b1
  #define SERIAL_RUNNING        0b10
 // #define SERIAL_RECONNECTING   0b001
  #define MS_BEFORE_TIMEOUT     1000
  #define SERIAL_INDEX_MAX      2 //2 is 16 bits, 4 is 32 bits, etc. 
  #define SEND_NONUNIQUE_MS     200
  //Marker characters
  /* take care not to set start or end marker to any character that has a binary ascii code that 
  includes a 1 in the 5th digit (xxx1xxxx) in it. For example, '(' is 00101000, but '<' is 00111100.
  This is because the 5th decimal place of the byte is reserved for acknowledgment characters. */   
  #define STARTMARKER   0x28 //'('
  #define ENDMARKER     0x29 //')'
  #define INITMARKER    0x67 //'g' from go
  #define ONEMSGMARKER  0x6F //'o' from onetime
  #define ACKMARKER     0x3D //'=' Recieved as expected, all good
  #define NAKMARKER     0x3F //'?' What's this supposed to be? Repeat
  //Data is recieved from pi in one 16 bit binary number. Bits defined as follows:
  // ---------------------------------------------------------------------------------------------
  //  |15|   |14|   |13|   |12|   |11|   |10    9|   |8|   |7    6    5|    |4|   |3    2    1    0|
  //E-Parity new? Unused reserved door 60mmFanSpeed  secL    BorrowLED   reserved     ReturnsLED
  // ---------------------------------------------------------------------------------------------
  #define POS_SERIAL_RETURN     0
  #define MASK_SERIAL_RETURN    0b1111 //4 digits
  #define POS_SERIAL_RES        4
  #define MASK_SERIAL_RES       0b1 //1 digit
  #define POS_SERIAL_BORROW     5
  #define MASK_SERIAL_BORROW    0b111 //3 digits
  #define POS_SERIAL_SECLIGHT   8
  #define MASK_SERIAL_SECLIGHT  0b1 //1 digit
  #define POS_SERIAL_FANSPEED   9
  #define MASK_SERIAL_FANSPEED  0b11 //2 digits
  #define POS_SERIAL_DOOR       11
  #define MASK_SERIAL_DOOR      0b1 //2 digits
  #define POS_SERIAL_NEW        14
  #define POS_SERIAL_EPARITY    15

Oh and funny that you mention that F() comment, I just write the word "documentation" so that I later remember to include something about it in my documentation.

That is certainly scary! What are the odds of those codes being in your binary data, somewhere?

I don't understand, what's scary? What codes? And does that have to do with my receive buffer not getting new characters?

Im just using a specific binary code to initialize, two specific ones to start and end a "packet", two specific ones to ACK or NAK, and the data can be anything that doesnt include the reserved bit or the start/end markers.

If its scary, what's the better way? This is all binary, I won't be receiving strings, only sending them.

Did you put code in the sending program to ensure your control characters never appear in the binary fields you are sending?

Yes. It uses the same definitions and does check each message before sending it. This is not my issue. Do you have any idea why the Arduino stops receiving charcaters into the buffer?

With some testing, I have determined that this is a bug, looks like specifically in USB serial. Tested both the Uno and the Leonardo with this code:

// Works perfectly on Uno (tested at 9,600, 115,200, and 2,000,000 baud). Doesnt work on Leonardo if two characters are sent.
void setup() {
    Serial.begin(0); //No baudrate for USB serial
}

void loop() {
    static uint8_t prevInBuffer;
    static uint32_t heartbeat;

    if (Serial.available() != prevInBuffer){ //Print a change in buffer, so we dont spam the monitor
        Serial.print(F("Buffer changed from "));
        Serial.print(prevInBuffer);
        Serial.print(F(" bytes to "));
        Serial.print(Serial.available());
        Serial.println(F(" bytes."));
        prevInBuffer = Serial.available();
    }
    else if (Serial.available() >= 4){ //bug: if a couple of characters are sent, buffer never updates, this condition is never satified.
        while (Serial.available()){
            Serial.print(char(Serial.read()));
        }
        Serial.println();
    }

    if (millis() - heartbeat > 1000){ //did the Arduino crash?
        Serial.print('*');
        heartbeat = millis();
    }
}

Some more testing revealed that the characters are received in the buffer, the issue is with Serial.available(); I submitted the following issue on github: USB Serial on 32U4 stops incrementing Serial.available() if serial buffer has ≥2 bytes in it and more are received · Issue #516 · arduino/ArduinoCore-avr · GitHub

You have a logic bug there as available() might have changed between the if test and when you print it and at the end when you save its value in prevInBuffer

Try

void setup() {
    Serial.begin(115200); // baudrate does not matter for USB serial
}

void loop() {
    static int prevInBuffer;
    static uint32_t heartbeat;
    int avail = Serial.available();
    if (avail != prevInBuffer){ //Print a change in buffer, so we dont spam the monitor
        Serial.print(F("Buffer changed from "));
        Serial.print(prevInBuffer);
        Serial.print(F(" bytes to "));
        Serial.print(avail);
        Serial.println(F(" bytes."));
        prevInBuffer = avail;
    }
    else if (avail >= 4){ 
        while (Serial.available()){
            Serial.print(char(Serial.read()));
        }
        prevInBuffer = 0;
        Serial.println();
    }

    if (millis() - heartbeat > 1000){ //did the Arduino crash?
        Serial.print('*');
        heartbeat = millis();
    }
}

If you send binary data xxx in between the parentheses (xxx) do you ensure that one of those x is not the ascii code for the closing parenthese ?

If you don’t Murphy’s law says it will happen at the worst time :slight_smile:

If you're talking about "Arduino", are you talking about the Leonardo or about the IDE?

A quick test with IDE 1.8.5 seems to indicate that Serial Monitor crashes and not the Leonardo.

  1. Send two bytes; expected behaviour.
  2. Send a third byte; Serial.available() does not see it.
  3. Send fourth byte; it's not cleared from the input box. Serial receive windows no longer updates with '*'. However, TX led still flashes at 1 second interval.
  4. The serial monitor window can't be closed till such time that the USB cable is removed or the Leonardo is reset.

The behaviour with IDE 2.0.3 is different in that '*' is still received and the input box is still cleared; (4) above does not apply.

Adding a blinking LED in the heartbeat also does show that the heartbeat is active so the Leonardo is not hanging.

I agree with that but it does not solve the problem :wink: Serial.available() should eventually be larger than 4 :wink:

I don’t have an 32u4 device with me. Did you test with the code I posted?