How to ensure serial data soesn't get lost ?

I'm currently working on a project, where serial data is "received" in data bursts. These bursts happen every few seconds and are about 100-200bytes long. I'm currently receiving at a rate 9600 baude. So far, so good - everything works. The problem starts, when the data is received while the display gets updatet. The display update-cycle takes 60ms which is 8times longer then the data-burst (7ms) and the data can't be read before the serial-buffer is overflowed.

Maybe i missed it, but i coulnd't find a "overflow" handler method/callback in the hardware serial implementation which would be invoked on an overflow.. What i (would) need is some interrupt (or callback) which has a higher priority and would interrput the display update if new data arrives.

Ok, i could modify the HardwareSerial to either invoke a callback or set a pin-interrupt...but i would appriciate a solution without modifying the original (ardiuino) code Any suggestions ?

You're looking in the wrong place - you need to break down you display update into shorter time slices.

AWOL: you need to break down you display update into shorter time slices.

don't think that possible - at least not to 5ms... (i'm using u8g with a 64x128 pixel display)

i'm currently experimenting with an overflow-call in store_char()

i'm currently experimenting with an overflow-call in store_char()

I wonder what that means.

What about a timer interrupt every xx ms to fetch a few waiting characters into a buffer

Even simpler is making the receive buffer bigger. Maybe we'll see some code, and cut out the guesswork.

AWOL:

i’m currently experimenting with an overflow-call in store_char()

I wonder what that means.

I modifyed the store_char() function from HardwareSerial.cpp (the else part)

void serial_overflow() __attribute__((weak));

inline void store_char(unsigned char c, ring_buffer *buffer)
{
  int i = (unsigned int)(buffer->head + 1) % SERIAL_BUFFER_SIZE;
  if (i != buffer->tail) {
    buffer->buffer[buffer->head] = c;
    buffer->head = i;
  } [color=red]else {
	  serial_overflow();
	  store_char(c, buffer);
  }[/color]
}

and here is my overflow handler… the buffer will be processed on the next loop() cycle

void serial_overflow() {
    while(Serial.available()) {
      buffer << (char)Serial.read();
   }
}

hope this shades some light into my sparse statement above

AWOL: Even simpler is making the receive buffer bigger.

yes, that would be the simplest solution. The code above doesn't work yet - just don't know why..might be, because i'm running out of memory - or because of an endless recurion (though i tought, the overflow_handler would "reset" tail/head) - or because i have a race condition in the overflow-handler

@knut_ny timers ? good idea - exactly what i was looking for....i'll give it a try.

The best thing to do would be to cut the lcd update in to smaller time slices - see blink without delay.

next best is to increase serials rx buffer space. But you have only 2k of ram so not a good idea.

You could also, reduce the size of the data burst, reduce the baud rate, speed up the writing of the lcd.

Mark

I had very same problem - updating lcd was slowing whole program. And this worked for me :

#define INTERVAL_LCD  30              
int lcd_counter = 0;           

setup() 
{
} 
loop() 
{
           
 if (( lcd_counter++ % INTERVAL_LCD) == 0 )                                       
  {
    u8g.firstPage();  
    do {
      draw();
    } 
    while( u8g.nextPage() );
  }  
}

You can change the INTERVAL_LCD - this is number of loops before lcd is updated. Higher number means slower update and more time to do other things in loop() .

@waki your Solution doesn't solves the problem. If the burst happens during the Rendering loop, the receive buffer will overflow.

So maybe it is possible to enter rendering loop Only When there is no data to read? Something like

if (! Serial.available() && lcd_counter++ % INTERVAL_LCD) == 0) 
{
Render loop
}

So maybe it is possible to enter rendering loop Only When there is no data to read?

no, this won't work either since there is now way to make sure, no data is sent during rendering

the only solution that will work, is to interrupt the rendering loop while a data burst happens and continue (the rendering loop) once the data has been fetched.

i'm trying the timer approach now - since this is the least invasive solution (compared to the overflow approach) but thanks anyway for your suggestions!

Jus a thought but where is the data coming from, can you use flow control?

MattS-UK: Jus a thought but where is the data coming from, can you use flow control?

I second Matt on this.

These bursts happen every few seconds and are about 100-200bytes long. I’m currently receiving at a rate 9600 baude. So far, so good - everything works. The problem starts, when the data is received while the display gets updatet. The display update-cycle takes 60ms which is 8times longer then the data-burst (7ms) and the data can’t be read before the serial-buffer is overflowed.

There is something wrong with your math!
Considering you will send 200 bytes @9600 bps to transfer all those bytes it will take 208 ms
Since each byte consists of 10 bits (start bit + 8 data bits + stop bit) we can transmit 960 bytes per second, so each byte takes 1.04 ms
1.04ms * 200 bytes(worst case) = 208 mS

The display update-cycle takes 60ms which is 8times longer then the data-burst (7ms)

So how can you get a burst frequency of 7ms?

wait...wait...you are right... its not [u]bytes[/u] per seconds (as i calculated) - but [u]bits[/u] :blush: and yes - ignored the start/stop/parity bits...

problem solved…I’m using a timer now to fetch the data. in short intervalls
To be honest - the original problem was caused by a mixture of low memory and bugs :wink:

@Khalid & MattS-UK
what to do you mean be flow control ? A protocol with flow controll (i.e. sending back ok before the next burst happends) ?
Or does the arduino provide hardware-flow controll ?

In my case i send data from computer to Arduino. I use special handshake protocol that tells arduino that a specific sized packet is arrived. Arduino send a confirmation byte back to PC which in turn send other byte to Arduino. In this way both happy without any communication loss. If you have serial.println function used in the arduino sketch then comments them out. You will see drastic change in your communication.