Is it possible to be 100% sure that no incoming data is being lost from a serial port?
I’ve written a library for processing GPS data, when I test it with simple test scripts it seems to work fine, when I incorporate it into a larger project with lots more functionality and dozens of other libraries then I’ve noticed that about 5% of the NMEA sentences received don’t match their checksums. My best guess at the moment is that I’m missing some of the incoming data because I’m not reading the Serial port often enough (because the arduino is busy doing other stuff).
To investigate this, I’ve modified my code to count how many bytes I read on each pass and flag a message whenever that number approaches the buffer size (128 bytes) – see following snippet....
//Check that there's data waiting to be read
while (Serial1.available())
{
//count the characters recieved
_bytesRead++; //(this variable is a byte datatype, 0-255)
//process current byte from the logger
if (encodeRecord(Serial1.read()))
{
//return true;
}
// record the most recent moment that some data has been recieved
_gps_data=millis();
}
This gave me results I didn’t expect. Since the buffer size is 128 bytes (I think) I was expecting the maximum number of bytes read (_bytesRead) to not exceed this number, however this value occasionally exceeds that. This is either a bug, or it indicates the arrival of extra data during the processing of the while loop.
To summarise my questions....
How big is the serial buffer on an Mega? I know it’s 128 on an arduino so I’m assuming it’s the same on the Mega?
Is it possible for a while(Serial1.available()) loop to process more than 128 bytes in one go? Or is this a sign of a bug in my code?
How does the Serial buffer work when it gets too full? I assume that when it overflows, the oldest few bytes gets pushed out one end whilst the arriving bytes get fed in at the other end (first in, first out). Or does it just flush the entire buffer?
Is there any way of telling when the serial buffer has overflowed and data has been lost?
Is it possible to be 100% sure that no incoming data is being lost from a serial port?
No. Reasonably certain, yes. Positive, no.
I’ve written a library for processing GPS data
You didn't like the TinyGPS or NMEA libraries?
This gave me results I didn’t expect. Since the buffer size is 128 bytes (I think) I was expecting the maximum number of bytes read (_bytesRead) to not exceed this number, however this value occasionally exceeds that. This is either a bug, or it indicates the arrival of extra data during the processing of the while loop.
It's the latter.
I know it’s 128 on an arduino so I’m assuming it’s the same on the Mega?
You can look in HardwareSerial.cpp to see, but it is not more than 128 bytes, on 0023 and earlier. The value ,ay be different on 1.0 and later. I haven't looked. The Arduino should be checking the ssslllooowww data stream from the GPS often enough to not matter.
Is it possible for a while(Serial1.available()) loop to process more than 128 bytes in one go?
Easily.
How does the Serial buffer work when it gets too full? I assume that when it overflows, the oldest few bytes gets pushed out one end whilst the arriving bytes get fed in at the other end (first in, first out). Or does it just flush the entire buffer?
Neither. If there is no room in the buffer for the new byte, it is discarded.
Is there any way of telling when the serial buffer has overflowed and data has been lost?
If Serial.available() returns 128, there may, or may not, have been data loss. Does it matter, though? That data is not recoverable.
My library is based on TinyGPS, but edited for my needs. Plus I learn more from hacking it myself.
If there is no room in the buffer for the new byte, it is discarded.
Thanks for that - that explains some of the data I'm seeing.
Does it matter, though? That data is not recoverable.
Maybe not, but I'd still like to know that I'm dropping data. - At least that way I've got some idea of wether I can trust it or not.
I've done some more work on this, it seems that Serial.available() is often around 120+, and the actual number of bytes read at the same time is often greater. My GPS is talking at 56k so data will be arriving at approximately 112 bytes every 2 milliseconds, so I guess I need to leave no more than that period between reads.
I think this figure refers to 5 NMEA sentences per second. Quoted baud rate is 56k.You're right, I've mixed up my units. 56k bits per second equates to 112 bits every 2ms, or 14 bytes/2ms. So a 128 byte buffer could fill in less than 20ms
We're picking nits, now, I know, but a baud rate of 56K bits per second only means that that is the maximum rate. It does not mean that the device WILL send 56K bits every second.
The real number of bytes per second that you need to deal with is 5 times the average length of a sentence, whatever that average length is.
Yes, you're right. The GPS unit isn't transmistting much data, but when it comes it comes fast.
I've done a bit of analysis, if I check the serial port about once every five milliseconds, then most of the time I recieve, no data. But every now and again I recieve a burst of data as the GPS transmits. This is what I see if I poll the port every 5ms and display the number of bytes available each time...
Perhaps it is time for you to upgrade to Arduino 1.0, then. There is now the ability to register a callback when serial data arrives. Instead of polling, you can ask to be notified (via callback) when serial data arrives.
BigusDickus:
Yes, you're right. The GPS unit isn't transmistting much data, but when it comes it comes fast.
...
_bytes_avail<18>,bytes_read<96>
_bytes_avail<70>,bytes_read<120>
If you poll at 5 mS intervals a GPS transmitting at 57600bps (5760 bytes per second), you won't get those figures. 1/5760 is a byte every 0.1736 mS, so to get 70 bytes would take 12.15 mS, not 5 mS.
bool MiniGPS::readGpsPort()
{
byte _bytesRead=0; // counts the number of characters received from the GPS each time it's read.
byte _bytesAvailable=0; // number of bytes available in the input buffer from the GPS
if ( _maxReadInterval <= (millis()-_readTs) )
{
// To long a period has passed between this read() and the previous one. Incoming data
// may have been lost.
Serial.print("ERROR - <");Serial.print(millis()-_readTs);Serial.println(">ms since data was last read. Serial data MAY have been lost.");
}
_readTs=millis(); // Timestamp the moment that the serial port has been checked.
// Record the amount of data available for processing. If this is equal to (or very close to) the
// buffer size of 128 bytes, then it may be that the buffer has overflowed and some data has been lost
_bytesAvailable=Serial1.available();
//Check that there's data waiting to be read
while (Serial1.available())
{
//count the characters recieved
_bytesRead++;
//process current byte from the logger
if (encodeRecord(Serial1.read()))
{
//return true;
}
// record the most recent moment that some data has been recieved
_gps_data=millis();
}
if (_bytesAvailable>=_buff_threshold )
{
// Input buffer close to it's limits, increase the sampling rate or
// risk losing data.
Serial.print("ERROR - <");Serial.print(_bytesAvailable,DEC);Serial.print(">bytes available, <");Serial.print(_bytesRead,DEC);Serial.print(">read. This exceeds the <");Serial.print(_buff_threshold,DEC);Serial.println("> byte threshold.");
}
if ( _no_data_threshold <= (millis()-_gps_data) )
{
Serial.print("ERROR - No data (of any sort) recieved from GPS for more than <");Serial.print(_no_data_threshold);Serial.println("> milliseconds.");
}
But the debug he's producing will seriously compromise his test results. It's going to take quite a while to produce the debug messages. Perhaps he should follow Paul's example and store the results in an array and display that when finished.
Hi Nick, the code that does the polling is also doing other stuff, the interval quoted was a minimum. When used it anger it's also sending messages and writing to log files etc etc. The exact interval is unlikely to be exactly that quoted.
There must be something in the air tonight. We seem to be begging people to post their code - the whole code.
Debugging messages can certainly affect the results, you need to be conscious that attempts to debug code by doing serial prints can actually make things worse.