Serial buffer overflow

system · February 26, 2013, 4:27pm

hello just want to ask, how can i detect a serial buffer overflow???

and if then my buffer is cleared or empty, how can i retrieve the dumped data ( am i right with my understanding here, that once buffer overflows, data are dumped to somewhere-i-don't-know in the serial device? if so can this data be retrieved? then, how am i going to re.initialize serial available command so as to continue the stopped portion of the data and get valid data again as arduino is receiving?)

i'm not really sure with this...i'm totally new to arduino...

i just want to achieve these things:

detect overflow
2.clear buffer
3.retrieve data from last (not really sure here)
4.get back to serial available command
start filling buffer again

anyways, main thing is i want to get a reliable serial communication ....and be able to recover once overflow happens.....

system · February 26, 2013, 4:31pm

If overflow occurs then data that was received could not be stored and is permanently lost.

The way to avoid overflow is to ensure that you read the serial data soon enough to avoid the receive buffer becoming full. That means designing your sketch so that it doesn't ever block (stop) for long enough to cause the receive buffer to fill.

It will still be possible for part of a message to become lost or corrupted so you will need to design your message protocol so that this can be detected and provide a means to recover if/when it happens. Normally it would be exceptionally rare for this to happen, though.

retrolefty · February 26, 2013, 4:40pm

and if then my buffer is cleared or empty, how can i retrieve the dumped data ( am i right with my understanding here, that once buffer overflows, data are dumped to somewhere-i-don't-know in the serial device? if so can this data be retrieved?

Well technically the serial library buffer management will not allow a true 'overflow' condition. Rather is uses a circular buffer, so if you fail to read characters out of the buffer in due time new characters coming in will simply overwrite the characters you failed to read in time. A 'robust' serial protocol can be designed to recover from such a condition and allow the sender and receiver to get back in 'sync', but the data lost is not retrievable.

Also one can always edit the serial library code to utilize a larger SRAM buffer size, but at the expense of less SRAM for users need.

Lefty

system · February 26, 2013, 4:43pm

i'm not really sure with this thing that's going in my system....

at a long period of time after buffer is filled with an incoming byte...and then parsed....there are times at which sending of data from serial device gets halted....is this overflow??? or overflow just means there will still be data left but not valid ...or there are portions of the data that are being skipped and thus, continues to just shift the remaining characters just for the compliance that buffer is filled with same exact number of bytes?

what i'm trying to say is, overflow still has data to display (garbage or anything that is invalid)? it does not completely mean that serial device stops sending data and halts serial display?

holmes4 · February 26, 2013, 4:47pm

Take a look at the code. On the tx side from the arduino you can't have over flow at all - Serial print just hangs.

Mark

system · February 26, 2013, 4:53pm

i'm sorry but i think i don't get it that much

am i supposed to get some thing to do with tx at the arduino side (my system is only receiving data from serial device, this is not transmitting whatsoever to a serial device) ???

how am i going to somehow put a solution to this hanging of serial.print??

and what might cause it????

system · February 26, 2013, 5:10pm

Try posting your code.
It has been notable by its absence so far in this thread.

retrolefty · February 26, 2013, 5:19pm

phoenix1118:
i'm sorry but i think i don't get it that much

am i supposed to get some thing to do with tx at the arduino side (my system is only receiving data from serial device, this is not transmitting whatsoever to a serial device) ???

how am i going to somehow put a solution to this hanging of serial.print??

and what might cause it????

Serial.print is a arduino transmit function so if your not sending anything then your not using serial.print statements? What you need to know is if your complete sketch is timely enough to keep up with the serial receiving data being sent to it.

We can't really help you without seeing your code and knowing something about the data stream being sent to your arduino.

Lefty

system · February 26, 2013, 5:28pm

Here:

#include <SoftwareSerial.h>
#include <LiquidCrystal.h>

LiquidCrystal lcd(9,8,7,6,5,4);

const int lcdLed=13;


char incomingByte;

int i;
int counter1=0;
int counter2=0;

const int rxPin=10;
const int txPin=11;
const int byteSize=53;

char * range;
char * watts;
char * va;
char * var;
char * pf;
char * volts;
char * amps;

char buffer[53];

bool startDataReceived=false;
bool dataAvailable=false;

SoftwareSerial mySerial(rxPin,txPin);

float wattHour,wattHour2,dummy,currentConsumption,kilowattHour;
float wattHour1=0;
float wattHour3=0;
float wattSecond;
float wattSecond1;
float wattSecond2=0;
float dummy1=0;
float lastConsumption=0;
float KWHInterval=1000; 

void setup(){
  pinMode(lcdLed,OUTPUT);
  digitalWrite(lcdLed,HIGH);
  pinMode(pulseLed,OUTPUT);
  digitalWrite(pulseLed,LOW);
  lcd.begin(16,2);
  Serial.begin(9600);
  mySerial.begin(9600);}
    
  
void displayData(){
 lcd.setCursor(0,0);
 lcd.print(wattSecond,0);
 lcd.print("w ");
 lcd.setCursor(0,1);
 lcd.print(wattHour/1000, 2);
 lcd.print("kwh ");
 lcd.setCursor(8,0);
 lcd.print(wattHour, 2);
 lcd.print("wh ");}

void storeKwh(){
   wattSecond=atof(watts);
   counter1=counter1+1;
   wattSecond1=wattSecond; 
   if(wattSecond2==wattSecond)
     wattSecond=wattSecond; 
   else{
     wattSecond=(wattSecond-wattSecond2)+(wattSecond2);
     wattSecond2=wattSecond1;
     wattSecond=wattSecond;}
   if(wattSecond==0)
     wattHour3=wattHour1;
   else{
     wattHour2=wattSecond/3600;
     wattHour=(wattHour3)+(wattHour2);
     wattHour1=wattHour;}
   if(counter1<=60&&wattSecond!=0){
     dummy=wattHour;
     wattHour=(dummy1)+dummy;
     dummy1=wattHour;}
   else{
     counter1=0;
     counter2=counter2+1;
     if(counter2<=60&&wattSecond!=0){
       dummy=wattHour;
       wattHour=dummy1+dummy;
       dummy1=wattHour;}    
     else
       counter2=0;
     currentConsumption=wattHour;
            if((currentConsumption-lastConsumption)>(KWHInterval)){
              lastConsumption+=KWHInterval;
              kilowattHour=(lastConsumption/1000);
              }}}
   

  
void loop(){
  
  if(mySerial.available()>0){
    incomingByte=mySerial.read();
    if(incomingByte=='I'){
      startDataReceived=true; 
      buffer[0]=incomingByte;}
      }
      
  if((startDataReceived==true)&&(mySerial.available()>0)){
    for(i=1;i<byteSize;i++){
      while(mySerial.available()==0){}
      buffer[i]=mySerial.read();}
    startDataReceived=false;
    dataAvailable=true;}
    
  if(dataAvailable==true){ 
    range=strtok(buffer, ",");
    watts=strtok(NULL, ",");
    va=strtok(NULL, ",");
    var=strtok(NULL, ",");
    pf=strtok(NULL, ",");
    volts=strtok(NULL, ",");
    amps=strtok(NULL, ",");
         
         storeKwh();
         Serial.println(wattSecond);
         Serial.println(wattHour,4);
         Serial.println(counter1);
         Serial.println(counter2);
         Serial.println(kilowattHour,4);

    
    memset(buffer,NULL,byteSize);
    dataAvailable=false;}}

system · February 26, 2013, 5:37pm

Please can you run the code auto-format tool on that, and repost it?
My eyes hurt.

system · February 26, 2013, 5:44pm

#include <SoftwareSerial.h>

char incomingByte;

int i;
int counter1=0;
int counter2=0;

const int rxPin=10;
const int txPin=11;
const int byteSize=53;

char * range;
char * watts;
char * va;
char * var;
char * pf;
char * volts;
char * amps;

char buffer[53];

bool startDataReceived=false;
bool dataAvailable=false;

SoftwareSerial mySerial(rxPin,txPin);

float wattHour,wattHour2,dummy,currentConsumption,kilowattHour;
float wattHour1=0;
float wattHour3=0;
float wattSecond;
float wattSecond1;
float wattSecond2=0;
float dummy1=0;
float lastConsumption=0;
float KWHInterval=1000; 

void setup(){
  pinMode(lcdLed,OUTPUT);
  digitalWrite(lcdLed,HIGH);
  pinMode(pulseLed,OUTPUT);
  digitalWrite(pulseLed,LOW);
  lcd.begin(16,2);
  Serial.begin(9600);
  mySerial.begin(9600);
}


void displayData(){
  lcd.setCursor(0,0);
  lcd.print(wattSecond,0);
  lcd.print("w ");
  lcd.setCursor(0,1);
  lcd.print(wattHour/1000, 2);
  lcd.print("kwh ");
  lcd.setCursor(8,0);
  lcd.print(wattHour, 2);
  lcd.print("wh ");
}

void storeKwh(){
  wattSecond=atof(watts);
  counter1=counter1+1;
  wattSecond1=wattSecond; 
  if(wattSecond2==wattSecond)
    wattSecond=wattSecond; 
  else{
    wattSecond=(wattSecond-wattSecond2)+(wattSecond2);
    wattSecond2=wattSecond1;
    wattSecond=wattSecond;
  }
  if(wattSecond==0)
    wattHour3=wattHour1;
  else{
    wattHour2=wattSecond/3600;
    wattHour=(wattHour3)+(wattHour2);
    wattHour1=wattHour;
  }
  if(counter1<=60&&wattSecond!=0){
    dummy=wattHour;
    wattHour=(dummy1)+dummy;
    dummy1=wattHour;
  }
  else{
    counter1=0;
    counter2=counter2+1;
    if(counter2<=60&&wattSecond!=0){
      dummy=wattHour;
      wattHour=dummy1+dummy;
      dummy1=wattHour;
    }    
    else
      counter2=0;
    currentConsumption=wattHour;
    if((currentConsumption-lastConsumption)>(KWHInterval)){
      lastConsumption+=KWHInterval;
      kilowattHour=(lastConsumption/1000);
    }
  }
}



void loop(){

  if(mySerial.available()>0){
    incomingByte=mySerial.read();
    if(incomingByte=='I'){
      startDataReceived=true; 
      buffer[0]=incomingByte;
    }
  }

  if((startDataReceived==true)&&(mySerial.available()>0)){
    for(i=1;i<byteSize;i++){
      while(mySerial.available()==0){
      }
      buffer[i]=mySerial.read();
    }
    startDataReceived=false;
    dataAvailable=true;
  }

  if(dataAvailable==true){ 
    range=strtok(buffer, ",");
    watts=strtok(NULL, ",");
    va=strtok(NULL, ",");
    var=strtok(NULL, ",");
    pf=strtok(NULL, ",");
    volts=strtok(NULL, ",");
    amps=strtok(NULL, ",");

    storeKwh();
    Serial.println(wattSecond);
    Serial.println(wattHour,4);
    Serial.println(counter1);
    Serial.println(counter2);
    Serial.println(kilowattHour,4);


    memset(buffer,NULL,byteSize);
    dataAvailable=false;
  }
}

system · February 26, 2013, 5:47pm

phoenix1118:
at a long period of time after buffer is filled with an incoming byte...and then parsed....there are times at which sending of data from serial device gets halted

If overflow occurs then you will continue to receive data but some of the bytes that were sent will be missing from the stream of bytes you receive.

It does not sound as if this is what is happening in your case, although it's not clear exactly what is happening.

Can you expand on the symptoms you're seeing?

Is the data being sent to the Arduino, or sent from the Arduino?

What device is the data being sent from or to?

What symptoms make you conclude that the 'sending of data from serial device gets halted'? (This is not a normal symptom of overflow; it may be that you're just describing it inaccurately, but perhaps there's something else going on such as a problem in the sender, or some sort of flow control being invoked.)

system · February 26, 2013, 5:47pm

i forgot to delete the void displayData()
anyways....just look at the relevant softwareSerial code.....

system · February 26, 2013, 5:54pm

yeah, i'm not really sure what to call it either...i mean the output....the printing of serial.print just hangs...

i'm not sure if it's an overflow, but judging from the comments answered by people here (considering that there should be at least characters that should be present but are just invalid, and not completely to a halt),i mean what really is the output is that after some time, SERIAL.PRINT HANGS....

anyways, i'm using 9600 baud rate as this is specifically required by the serial-outputting device i used (i used power analyzer that outputs csv format)

the DATA IS SENT TO ARDUINO....arduino just receives it.....

system · February 26, 2013, 6:13pm

anyways here is the serial monitor output

Serial.println(wattSecond);      //89.80
Serial.println(wattHour,4);      //74.1634
Serial.println(counter1);         //59, this is every 60 seconds
Serial.println(counter2);         //46,this is every 60 minutes
Serial.println(kilowattHour,4); //0.0000

The numbers above correspond to the last values before serial.print hangs....

so....the serial.print hangs right after 46 minutes and 59 seconds....

and i did some other test too....
second attempt, it hangs at 1 hour and 29 minutes
third attempt, it hangs at 33 minutes

time here is obtained using counter

system · February 26, 2013, 6:14pm

system · February 26, 2013, 6:15pm

please...please...
someone help.....

system · February 26, 2013, 6:28pm

I see that the sketch reads a fixed size message from a SoftwareSerial, processes it to update some internal variables and then prints the current state of those variables.

The code reading from the software serial port isn't trivial, but I can't spot anything wrong with it. The most likely explanation I can see is that this condition is not being met:

if(mySerial.available()>0){

To see whether that's the case I suggest you write a new sketch that does nothing but initialise the hardware serial and software serial interfaces, and prints everything received on the software serial to the hardware serial. That would produce a minimal sketch that ought to show the same symptoms.

Also, look at the device at the other end of the software serial connection (I don't think you've told us what it is) and look for possible reasons why it would stop sending. The most likely reason that mySerial.available() would return zero is that nothing is being sent.

Finally, it would be a good idea to post a drawing or picture showing what you have connected and how. It's possible it's something silly like missing ground connections.

robtillaart · February 26, 2013, 7:35pm

while(mySerial.available()==0){ }

This is the blocking loop which freezes your system

I changed my (hardware) Serial code the following way (with some mods in the .h file too) to detect overflow.

struct ring_buffer
{
  unsigned char buffer[RX_BUFFER_SIZE];
  int head;
  int tail;
  unsigned long lost;  //<<<<<<<<<<<<<<<<< added this
};

#if defined(UBRRH) || defined(UBRR0H)
  ring_buffer rx_buffer  =  { { 0 }, 0, 0 };
#endif
#if defined(UBRR1H)
  ring_buffer rx_buffer1  =  { { 0 }, 0, 0 };
#endif
#if defined(UBRR2H)
  ring_buffer rx_buffer2  =  { { 0 }, 0, 0 };
#endif
#if defined(UBRR3H)
  ring_buffer rx_buffer3  =  { { 0 }, 0, 0 };
#endif

inline void store_char(unsigned char c, ring_buffer *rx_buffer)
{
  int i = (unsigned int)(rx_buffer->head + 1) % RX_BUFFER_SIZE;

  // if we should be storing the received character into the location
  // just before the tail (meaning that the head would advance to the
  // current location of the tail), we're about to overflow the buffer
  // and so we don't write the character or advance the head.
  if (i != rx_buffer->tail) {
    rx_buffer->buffer[rx_buffer->head] = c;
    rx_buffer->head = i;
  } else rx_buffer->lost++;  //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< filled here
}

.....

uint32_t HardwareSerial::overflow(void)  // <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< added this function.
{
  return _rx_buffer->lost;
}

hope it helps

Topic		Replies	Views
Arduino serial RX overflow Programming	30	6156	May 5, 2021
HELP -> 64 bytes buffer question Programming	22	5268	May 5, 2021
Receiving a large line by Serial port. Buffer problem Programming	17	5585	May 5, 2021
Sending command over serial >64 bytes (>128 bytes) Networking, Protocols, and Devices	18	18039	May 6, 2021
Can't Avoid Serial Buffer Overflow! Networking, Protocols, and Devices	13	22402	May 6, 2021

Serial buffer overflow

Related topics