Serial.available misbehaving on Leonardo

I’m using an Arduino Pro Micro, based on the Leonardo, for serial communication.

I’m finding the Serial.available command is misbehaving, as can be seen in the following example, which just lets the buffer fill with incoming characters, and reports how many there are every second:

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  Serial.println(Serial.available());
  delay(1000);
}

If you open a serial terminal, the program steadily prints 0s, as you would expect.

Send a character to the Arduino, and it returns 1s. So far so good.

Send another character and it continues sending 1s. Strange, it should be sending 2s now.

Send a third character and the serial terminal crashes, along with the entire Arduino environment, until I unplug the USB connection to the Arduino.

This is baffling, since I’m sure I have used the serial buffer on the Leonardo before. Could it be something to do with the upgrade to Windows 10? I’d appreciate anyone’s help. Thanks.

Strangely the problem above seems to have fixed itself. The buffer now fills up to 63 bytes, as expected. Restarting the Arduino environment may have fixed it. I didn't restart my PC.

The Ardunio terminal program still crashes in the same way when you overfill the buffer beyond 63 bytes, so my original problem seems to have been that the Arduino was behaving as though its buffer length was 1.

Any tips still appreciated.

The Ardunio terminal program still crashes in the same way when you overfill the buffer beyond 63 bytes

Nonsense. The "Arduino terminal program" has no idea whether the buffer on the Arduino is full.

so my original problem seems to have been that the Arduino was behaving as though its buffer length was 1.

The problem is that you didn't share your code. The Arduino KNOWS that the buffer size is not one.

That's not a very helpful reply. I did share my code, and described exactly what's happening.

Anyone (else) have an idea what might be happening here? Since my last post, the situation described in my first post has returned. Restarting the Arduino environment hasn't helped this time. Perhaps it's some problem between the way the Leonardo emulates the serial port, and the windows drivers that interface with it? Has anyone seen this before?

The examples in Serial Input Basics are simple reliable ways to receive data. They empty the input buffer faster than it can fill.

...R

Thanks I appreciate that, but there are times when reliable buffering is really useful; when receiving data in packets for example. The problem I'm facing here isn't that the buffer is filling up (64 bytes is plenty), it's that on occasion it doesn't seem to be performing its function at all, and I don't know why.

I'll keep investigating, but I wonder if anyone else is familiar with this kind of problem or if it's just a quirk of my setup?

paulrd: Thanks I appreciate that, but there are times when reliable buffering is really useful;

I don't know if this refers to Reply #5?

I don't believe there is anything unreliable about the buffering in Serial Input Basics. If you have found a problem I would like to know so that I can fix it.

I don't know what you mean by

isn't that the buffer is filling up (64 bytes is plenty), it's that on occasion it doesn't seem to be performing its function at all

What do you mean by "it" and what do you mean by not "performing its function"? What should it do, and what does it do?

...R

Hi Robin, and thanks for the time you have taken to share your serial examples.

First I'll describe my problem in more detail:

The problem appears to be unique to Leonardo-based Arduinos, i.e. those in which the AVR itself is used to provide the USB interface, but which appear as a COM port on the PC, like any other Arduino.

Sometimes, and I haven't established what causes this to start or stop happening, the Arduino keeps receiving bytes from the PC into its serial buffer, but the number returned by Serial.available() never advances above 1. The example code I gave in my first post demonstrates this.

This means that an Arduino program that waits to receive, for example, 4 bytes before parsing them, will stop reading bytes from the serial buffer, because Serial.available() >= 4 will never be satisfied, even when the bytes have been received. This is what I mean when I say it (the buffer) is not performing its function.

When this happens, and the PC keeps sending data, (usually only 1 or 2 bytes more), this in turn causes the PC serial terminal to stop responding. I've witnessed this with both with the Arduino terminal, and Termite. The programs freeze completely until the Arduino is unplugged, or the Arduino reads the waiting bytes from the buffer. I don't think this could happen with the FT232-based Arduinos, because the sending PC receives no information about what's happening in the Arduino's buffer, but as I said above, perhaps some interaction of the USB routines running on the AVR and the PC's USB-serial driver is causing the lock-up. I don't know.

As for your code...

I see from your code that you create your own 32 byte buffer for received serial characters: receivedChars[]. You check for newly arrived characters using Serial.available() > 0 and read them straight into the second buffer.

I think your method would help with the problem I describe above, as long as it is called frequently enough. It means there is no need for the Arduino's serial buffer to hold more than one byte, and Serial.available() is never required to return the true number of bytes in the buffer, as long as it returns >0. But it is a workaround rather than a solution, and it seems a shame to have to implement a second serial buffer in a program to mask the fact that the 64 byte buffer that already exists isn't working.

I hope this makes everything clear. If I can provide any more information please let me know.

paulrd: It means there is no need for the Arduino's serial buffer to hold more than one byte,

My code does not require to be called every time there is 1 byte available in the buffer. It will take all the characters that are currently in the buffer. (It used to just read one at a time but I amended it).

I am not aware of the problem that you describe because I rarely use my Leonardo. I agree it could be a PITA. I have an application that deliberately waits until all the data from the PC is in the input buffer.

However I think you could easily adapt the code in Serial Input Basics so you could see how many characters it had received even if it had not yet received the terminating character. You will eventually need to take the data out of the input buffer in order to use it so I don't think there is any wasted activity.

...R

paulrd: This means that an Arduino program that waits to receive, for example, 4 bytes before parsing them, will stop reading bytes from the serial buffer, because Serial.available() >= 4 will never be satisfied, even when the bytes have been received. This is what I mean when I say it (the buffer) is not performing its function.

What about the sending application on the PC?

Is the PC application possibly opening and closing the Serial connection all the time? - opening the Serial connection - sending bytes - closing the Serial connection And doing this again and again and again?

Or is the PC application behaving like a terminal program - opening the Serial connection - sending bytes - pause - sending bytes - pause - sending bytes - pause - and so on And never closing the Serial connection after sending some bytes?

Hi,

Jurs, the serial programs I've used all keep the connection open.

Robin2, I agree I could write a workaround like you describe, but it would be a shame to have to, as I'm trying to keep my code simple and compatible with all Arduino types, most of which have no need for a fix. I'd rather get to the root of what's causing the problem in the Leonardo, as it seems to be a flaw either in the bootloader code or the windows driver. Then maybe it can be fixed for all of us.

So far, I can't tell what causes the problem to come and go as it does. I'll keep trying to work this out so at least it's repeatable.

paulrd: Jurs, the serial programs I've used all keep the connection open.

I'd say it is caused by the CDC serial driver on the PC then.

If possible, try a different PC with a different operating system and compare, whether the same problem arises with that hardware, too.

I've made some progress with the problem described above.

I've discovered that when the problem is manifesting itself, I can still send multiple bytes to an empty buffer, as long as I do it in a single operation. By 'operation' I mean a single click of the 'Send' button in a serial terminal, or a single serial command in a programming language. The bytes arrive safely, and the number returned by Serial.available() is correct.

However, if I send bytes in separate send operations, I can only carry out 2 sends without reading the buffer in the Arduino; the 3rd will cause it to freeze until the buffer is read.

My best interpretation of what's happening is that the Leonardo has two 64-byte input buffers, but each one can only contain a single write operation. So sending 3 bytes in 3 separate operations causes a crash, but sending up to 63 bytes in a single operation up to twice is fine.

So to avoid problems, bytes intended to be read together in the Arduino must be sent together from the PC; sending them separately, even in quick succession, causes them to end up in different input buffers, and Serial.available() will only return the number of bytes sent in the first operation, potentially confusing your receive program.

I'm glad to have made some headway. As for why this problem only presents itself some of the time, that's still a mystery. :confused:

My best interpretation of what's happening is that the Leonardo has two 64-byte input buffers, but each one can only contain a single write operation. So sending 3 bytes in 3 separate operations causes a crash, but sending up to 63 bytes in a single operation up to twice is fine.

That is not true. The serial interrupt has NO idea how many bytes were sent together. All it knows is that a byte is here, ready to be read and put in the incoming serial buffer.

You clearly have problems with code you are not posting.

PaulS, apparently with a Leonardo, the way data is distributed among USB packets does affect its behaviour. I'm not familiar with the exact way the Leonardo handles the USB connection, but from your responses it's very clear you aren't either, so I think it would be best if you were to leave the discussion to people who are.

If the behaviour of the Leonardo depends on how many "sends" are done on the PC it seems very likely that the PC is doing something that you are not aware of. Perhaps it is asserting or deasserting one of DSR DTR CTS etc.

Try writing a Python (or your favourite language) program on the PC and using it to send the data. That will give you full control of everything and will be a more reliable diagnostic instrument.

Have you tried another PC with a different operating system?

Another test might be to connect the PC to Tx1 and Rx1 using an FTDI cable as that would be using a different USB-TTL converter.

...R

Robin2: Another test might be to connect the PC to Tx1 and Rx1 using an FTDI cable as that would be using a different USB-TTL converter.

Not really. In case of Leonardo there is NOT a virtual serial port on the PC communicating with a real serial port on the Arduino.

BOTH serial ports are virtual.

PC: virtual CDC serial port Arduino: virtual CDC serial port

There is no real serial port involved, all serial communication is handled with CDC virtual USB drivers, on both ends of the communication.

Therefore it is essential, that all communication handling occurs as it should occur.

For example, also the LEONARDO is doing a auto-reset, when the serial connection is established, because the USB cable is plugged in.

BUT: Other than with Arduinos that have a real hardware serial, the CDC USB serial driver needs a short moment before Serial on the LEONARDO becomes a valid serial port.

So before using Serial, you should possibly - first wait for Serial to establish the communication driver - then use Serial to send and receive

For example with the initially posted program (while-loop in setup() added):

void setup()
{
  Serial.begin(9600);
  while (!Serial); // wait for serial port to connect. Needed for native USB
}

void loop()
{
  Serial.println(Serial.available());
  delay(1000);
}

What about that? Waiting for Serial to become active, then start to use it? Any changes?`

jurs: Not really. In case of Leonardo there is NOT a virtual serial port on the PC communicating with a real serial port on the

I think we are at cross purposes.

I imagine the relationship beween the PC and the 32U4 is the much same as the relationship between the PC and the FTDI chip in an FTDI cable.

I was just suggesting a way to see if the problem is peculiar to the 32U4 or is more general.

...R

I have been following this thread with interest but could not be bothered to find my Leo to try it. Today, whilst ferreting out some hardware for something else I came across the Leo so I tried it out.

For me (Windows 7/IDE 1.5.6-r2) the program in the original post does exactly what I would expect. It does not stall at any number bytes input until it reaches 63, which is what I would expect.

Has anyone else apart from the OP tried it on Windows 10 ?

I have only experienced the problem a Pro Micro, which uses a 32U4 and the Leonardo Firmware. Models that use a separate USB-serial converter don’t exhibit the problem, and I wouldn’t expect them to, given that the serial bytes are never buffered for long, always being passed on immediately to the AVR as a serial stream.

Today I tried my Pro Micro on a friend’s Windows 10 laptop, and found the same problem occurred. So it’s looking likely to me that the problem is the windows 10 CDC drivers, and their interactions with the Arduino Leonardo firmware.

Robin2, in answer to your earlier question, all communications have been with DTR active as far as I know. In my experience Leonardo based Arduinos refuse to communicate at all without this being the case. Jurs, I know the ‘while (!Serial);’ is missing from my original post, but it doesn’t make any difference in this case.