Serial output buffer clear/reset (Leonardo USB CDC specific?)

Hi all, I have an odd bug with the Leonardo USB, and cannot find any similar issue - so I guess I'm doing something wrong.

It's very simple - if the USB connection closes dirty (cable pulled or in this case, the host resets), my Leonardo doesn't see this and will continue to 'send' data.

However, something know's its not connected, as it'll buffer the data until a connection is restored.

This buffer will continue to fill on each send, soon causing a lock-up (presuming an overflow) needing a manual reset. Observable is the send/receive LED's - send becomes constantly lit.

Simplified snippet for anything serial: NOTE: assume a timer or a button calls SerialPrint() with a string of "Hello World".

void setup() {
  Serial.begin(9600);
}

void loop() {
  //This part polls Serial.available() for data. This works fine.
}

void SerialPrint(String str) {
  if (Serial) {
    Serial.println(str);
  }
}

This works fine in general, data in, data out.

If the connection breaks correctly, the 'if (Serial)' appears to behave correctly. (No send light). If the connection breaks dirty, the 'if (Serial)' remains true, so will call Serial.print. (Send light stays lit).

I have tried:

Serial.flush(); Before the print immediately, and also after with a (interrupt) delay of 10 seconds.

Is there a way to detect/clear the buffer?

I don't 'really' wish to modify the hardware to reset. At worst, a "asm volatile (" jmp 0");" may be doable - but ideally I never should need to reset.

Other important info?: - All Leonardo hardware interrupts are in use.

Thanks for reading!

Hi, welcome to the forum.

The bootloader of the Leonardo was changed with version 1.0.3 in 2012. http://www.arduino.cc/en/Main/ReleaseNotes

I have my Leonardo with bootloader of version 1.5.8. I think I have something else than you have.

Do you have a power supply to the Leonardo ? So it stays on when the usb is disconnected ?

When I disconnect the USB, and reconnect it, the serial monitor of the Arduino IDE does not continue.

This is my test sketch. I should add something else to the Leonardo, like a display or so. Just this sketch doesn't give me enough information.

// Tested with Arduino 1.6.0 in Windows 7
// Arduino Leonardo R3 with bootloader of 1.5.8
//
// Arduino Leonardo also powered via DC barrel jack.
//
// Pin 2 to ground is keep sending text
// Pin 13 is status of Serial

unsigned long count = 0UL;   // 0UL is Zero-Unsigned-Long

void setup()
{
  pinMode(13, OUTPUT);

  Serial.begin( 9600);
  ShowSerialStatus();

  // for Leonardo , wait to open serial monitor and a timeout of 15s
  unsigned long prevMillis = millis();
  while ( !Serial)
  {
    if ( millis() - prevMillis >= 15000)
      break;

    ShowSerialStatus();
  };

  if ( millis() - prevMillis >= 15000)
    Serial.println( "Continued after timeout");
  else
    Serial.println( "Continued after opening serial monitor");

  pinMode( 2, INPUT_PULLUP);
}

void loop()
{
  if ( digitalRead( 2) == LOW)
  {
    Serial.print( "Hello world");
    Serial.print( ", count=");
    Serial.println( count++);
  }

  ShowSerialStatus();

  delay(50);        // slow down the loop, but not too slow
}

void ShowSerialStatus()
{
  // Show 'Serial' to the system led.
  if ( Serial)
    digitalWrite( 13, HIGH);
  else
    digitalWrite( 13, LOW);
}

Do you have a test sketch for me, and a step-by-step guide that I can do to test it ?

Hi, thanks for the reply. Sorry for not being as verbose as I could have been, so feel free to skip over anything unrelated:

It's 'real world' setup is as follows: Leonardo is powered by a 12v supply with battery backup. It connects to a raspberry pi via USB and shares the PSU. PSU is 5A, filtered, with LA battery as mentioned - so quite capable.

Side note: The Pi is the most sensitive to the power supply, thus has been very well tested.

The issue for usb disconnect is simply a bug in the software running on the Pi. Despite 'that' being fixable, the arduino sketch is written to work regardless for fail-over.

Flow:

Event on Leonardo will send a message over serial. So pin toggle will send message.

Pi sends message -> Leo handles request -> event sends message to Pi.

or

Button pressed connected to Leo input -> Leo handles request -> event sends message to Pi.

If the Pi decides to close the port (this can be random - often weeks between failures, but can be forced), the Leo doesn't get a close event as it would if the port closed cleanly. Leo checks Serial, it's true, sends message.

I can repeat with bare bones code: blink an LED, send large but reasonable sized string via serial often, observe send LED blink on send, break USB connection, observe the send LED hang, blink LED will soon hang.

I do not know the bootloader version - so I'll try a sketch out that should report this, then attempt to find(!) the latest if too old.

As there has been no comment of 'you forgot to do this', I'm guessing it is likely a firmware issue and not my misunderstanding. So I'll do a hardware reset if no serial reply (naturally no reset if serial is closed) - bodgy, but at least will preserve the fail-over.

Thanks.

DubKat: Button pressed connected to Leo input -> Leo handles request -> event sends message to Pi.

If the Pi decides to close the port (this can be random - often weeks between failures, but can be forced), the Leo doesn't get a close event as it would if the port closed cleanly. Leo checks Serial, it's true, sends message.

I wonder if this could be managed with some smarter code. Suppose the Leonardo only sends data after it receives a request from the PI Then if the PI disconnects the Leonardo won't send anything When the PI reconnects it should read and discard all the stuff in the PI's input buffer and send a new request to the Leonardo.

...R

The Pi is not a computer, they might behave different when disconnecting the USB. When I press the reset button of the Leonardo (instead of turning off the power) I get a whole different kind of things with or without serial monitor open. Very confusing.

The test with blinking a led is not enough for me. I have to know more.

The "if (Serial)" during runtime is not fail-safe in my test. I think that confirms what you have ? I don't know if any status of the CDC serial port can be requested.

You could implement it in software with some kind of keep-alive, or handshake, or a request to the Pi that the Pi must respond to. But if the Leonardo would really be halted, it won't work of course. (While I was working on this reply, Robin2 also wrote about fixing this in software).

My conclusion: You try to communicate between two unpredictable parts (Leonardo CDC USB and Raspberry Pi). That can't be good, eliminate one unpredictable part. Use an Arduino Uno or a seperate usb-serial converter on the Leonardo. Or good old serial RX/TX communiation (with level shifter), or I2C.

You are both on the ball - so here's some info that I didn't think of:

  • The Pi is the part of the main project, but should be ignored for the scope of this issue:
  • The issue still occurs if the USB is removed/pulled from a laptop.
  • The issue occurs on any PSU.

  • The serial send occurs on events, not necesarily from serial requests. Simplified scenario:

  • If button A is pushed, call 'ToggleTest()', this will toggle an LED and send a serial message to say the LED is ON/OFF.

  • If button B is pushed, send a serial message to the Pi.

  • If the Pi sends a message to the Leo, call 'ToggleTest()'

I'll re-do my test code if needed, but really is as simple as the above.

For context (note: do not take this in to consideration, and do not worry about safety implications - there are safety overrides elsewhere) The 'real' code logs a keypad and door latch. The Leo has a fail-safe that, if it cannot communicate with the Pi, and will respond to a master key code stored in the Leo. The door also has a release button internally that the Leo will interrupt on and open the door - the only interaction with the Pi is to 'log' it. Currently, until the lock-up is figured, I have a temporary external button to trigger the release (parallel to the int button), meaning that I all the Leo does is send serial messages on entry/exit. The Pi can hang at anytime, so the Leo's release button messages can collect up until it finally hangs (typically during the release cycle!).

The bench test is purely:

Laptop, Leo, USB lead, PSU, LED's, test sketch - nothing electromagnetic, no weird libs, etc. :)

Fix #1 I'll likely aim towards is a response mechanism: Leo sends a serial message, then starts a watchdog timer. No response == WTD timeout reset. Response == WTD disable.

This will only work if the watchdog doesn't share an interrupt with the the pin interrupts. But once reset, it is happy that no serial connection is available and will work forever in this state.

Fix #2, As above, but a regular timer, pulling a pin jumpered to the reset pin.

As the serial buffer can hold a good few messages (way over required), and none are important, so no need to cache them if no connection available, I can tolerate the reset.

Dream fix was to detect the buffer and either prevent further Serial sends, or to clear or reset the Serial lib.

Alternate fix is to use another Arduino with a minimum of 4 interrupts AND serial - the Leo was preferred as it had USB, meaning I'm not limited to the Pi or using serial adapters. Pretty much as you suggest Peter.

Thanks!

Perhaps the USB trouble on the Pi would be fixed, but suddenly it happens once a year. That is still not acceptable. Also resetting the Leonardo to initialize the USB CDC serial again does not feel right. There are too many uncertainties. If it was my project, I would definitely not continue this way, and use RX/TX serial or buy a simple usb-serial module as I wrote before. When it is only logging and a few simple commands, even SoftwareSerial with a simple usb-serial module would work.

For logging are many possibilites. The Arduino could log to EEPROM, and the Pi could request the newest logs (from EEPROM). Or with a (micro)SD module.

DubKat: The Pi can hang at anytime, so the Leo's release button messages can collect up until it finally hangs (typically during the release cycle!).

That's why I suggested the Leonardo should only send data in response to a request from the PI. The Leonardo could store in (RAM or EEPROM) some minimal data to identify messages it is due to send but has not yet been asked for. It does not have to store the text of the messages.

In any case the messages should be in "code" so they can be as short as possible. 255 different circumstances could be encoded in a single byte. If you want to restrict yourself to readable characters for simplicity there are still more than 62 options using a single character. Let the PI do the hard work.

The PI could request data regularly (every few seconds, perhaps) and the Leonardo might normally just reply with NIL.

...R

Peter_n: The Pi is not a computer,

Really ...?

...R

Peter_n: Perhaps the USB trouble on the Pi would be fixed, but suddenly it happens once a year. That is still not acceptable. Also resetting the Leonardo to initialize the USB CDC serial again does not feel right. There are too many uncertainties. If it was my project, I would definitely not continue this way, and use RX/TX serial or buy a simple usb-serial module as I wrote before. When it is only logging and a few simple commands, even SoftwareSerial with a simple usb-serial module would work.

Exactly, which is why I wanted to know if there was a smarter way, but sadly not. The (real?) RX/TX pins are unfortunately shared with two of the interrupt pins, which are needed. If I could SoftSerial on some spare pins, I'll probably try this next.

Robin2: That's why I suggested the Leonardo should only send data in response to a request from the PI. The Leonardo could store in (RAM or EEPROM) some minimal data to identify messages it is due to send but has not yet been asked for. It does not have to store the text of the messages.

Maybe, I'm not overly keen on polling systems - Not quite as speedy/efficient, but entirely workable, polling every 250ms. The 'text' is built anyway for the serial packet - formatted into a JSON string. The data itself can be stored in an array and then the packet built on demand. (it'll be something like '1,987654321' on memory) The array will flush after 2-3 seconds anyway (no need for a key pad entry to be stored - we don't need a mornings worth of unlocks to flood the rest of the system.)

Thanks for the ideas!

Robin2: Really ...?

Okay Robin2, it is a computer and can run Windows 10, linux, the works.

(Don't be tempted to mention Apple Works now)

DubKat: I'm not overly keen on polling systems

I can't imagine why not. They work very well and it seems to be the perfect solution for your problem

...R

Robin2: I can't imagine why not. They work very well and it seems to be the perfect solution for your problem

...R

In this case, it'll work - but it's a bodge, certainly not perfect. It's akin to me asking why every time my car hits 60Mph - the engine cuts out, and being told the 'perfect solution' seems to be to take the bus.

But, for now, the bus is still a solution, so I'm not going to complain.

Polling in general: it's unnecessary overhead (extended memory usage on low memory devices isn't desirable, extended processing - handling, storing, recovering, sending instead of send or handle, send), it's not event driven for time critical setups. Setting flags for data in buffer is better, but still a kludge when unneeded.

(While I'm happy to do the same thing on a Cortex, it seemed a little overkill for what I feel even a Leo was overkill for! I'm guessing the CDC serial is really beyond the scope of access without some Bootloader tweaks at least.)

Using a register inside the microcontroller too often is also overhead ? :smiley_cat: Looking at your watch to see the time is polling, is that overhead ? :confused: I agree with Robin2, polling can be the right thing to do in a certain situation, nothing wrong with that. 8)

Peter_n: Using a register inside the microcontroller too often is also overhead ? :smiley_cat: Looking at your watch to see the time is polling, is that overhead ? :confused: I agree with Robin2, polling can be the right thing to do in a certain situation, nothing wrong with that. 8)

I guess the string truncation in reading my comment proves my point :D

DubKat:
In this case, it’ll work - but it’s a bodge, certainly not perfect.

I have no wish to cause an argument but I am genuinely interested to understand why you say that and also to understand the type of solution that you would consider to be good.

…R

Robin2: I have no wish to cause an argument but I am genuinely interested to understand why you say that and also to understand the type of solution that you would consider to be good.

...R

Of course, neither do I. I also appreciate different backgrounds have a greater use for different methods, as different situations require. I also understand polling is still fundamental in very high speed/high data systems. Especially when shifting data from high to low speed.

Without writing a wall of text, I prefer it in this situation as it squashes a handful of potential issues (interrupts potentially jumping halfway through a serial write) and keeps the footprint smaller for future expansion, as well as keeping it simple to read for me or others that take the project in months or years time.

As per my original post, the Leo is already doing the heavy lifting, but just needs that 'spotter' to keep check.

My alternative now is to flip your idea around - rather than buffer and send on request, just acknowledge receipt:

1 - Pi connect's, sends an 'ACK'. 2 - Leo see's 'ACK' and sets flag. 3 - On next event, check flag. If true, send, then clear flag. 4 - Pi sends 'ACK' on receipt. 5 - Goto 2.

If the Pi is stolen or explodes, corrupts its SD - at any point, the Leo still tries to send the packet once, but never again and will never overflow (assuming no bugs in the CDC code).

Interestingly, from CDC.cpp

         /* only try to send bytes if the high-level CDC connection itself
     is open (not just the pipe) - the OS should set lineState when the port
     is opened and clear lineState when the port is closed.
     bytes sent before the user opens the connection or after
     the connection is closed are lost - just like with a UART. */

    // TODO - ZE - check behavior on different OSes and test what happens if an
    // open connection isn't broken cleanly (cable is yanked out, host dies
    // or locks up, or host virtual serial port hangs)

Looks like I've hit that 'todo' :S

DubKat:
My alternative now is to flip your idea around - rather than buffer and send on request, just acknowledge receipt:

Yes, that should deal with the situation perfectly well.

The system I suggested arose from an application where the PC needed to get the answer quickly (or a timeout) so it could do something else. Horses for courses.

…R