Serial1.available() and Serial1.read() does not handle zero (0x00)

Do you have a link to the Arduinos that you are actually using? I've never heard of a 3.3V Mega2560.

Typically, 3.3V Arduinos run at lower clock speeds than 5.0V Arduinos. The slower clock speed means that 115200 is iffy. It would be like trying to run a 5V device at 230400.

PaulS,
MEGA2560 Pro 3.3V from Sparkfun.com
Part# DEV-10744

sixeyes,
your test data works fine under 115200.

Is this it?

That runs at 8 MHz, compared to 16 MHz for the normal Uno/Pro.

Pity you didn't mention that initially. Running at "half speed" compared to what we would expect it to be running at, it isn't entirely surprising that the fastest baud rate isn't supported.

Let's take a look...
http://www.wormfood.net/avrbaudcalc.php

8 Mhz ... 115200 ... 7.8% error ... That is definitely in the problem range.
8 Mhz ... 57600 ... 3.7% error ... Much better!

I recall that anything over 10% error is fatal. Problems arise when the two clocks are a bit off or when the bit pattern is not broken up (the UART resynchronizes on a bit edge).

For what it's worth, I've had good luck keeping the clock error on each side below 4.5%.

And your initial posted sketch:

hanlee:
Here is the simple schetch:

void setup()
{
Serial.begin(9600);
Serial1.begin(9600);
}
...

You now say that it works at this speed. So just for future reference, when posting problems:

  • Post the sketch that causes the problem, not a "similar" one that doesn't.
  • Describe your hardware setup exactly
  • Post any other information needed to reproduce the problem, in your case the "sending" sketch

Oh, and how about this?

I asked this:

Did you have both input and output at 115200? Or were you reading at 115200 and displaying debugging at 9600?

You replied:

Of course I've paired same baud rate.

Well, no you did not do that!

hanlee:
Here is my sketch of the receiver:

void setup()

{
  Serial.begin(9600);
  Serial1.begin(115200);
}
...

I guessed you were reading at 115200 and debugging at 9600 and that is exactly what you were doing!

So when you said "Of course I've paired same baud rate" - that was just plain wrong. I asked if you were reading at one speed and debugging at another. That was the question. Of course you "pair the baud rate" between devices. But that wasn't the question.

This is very frustrating when you don't read what we are suggesting and don't post your code until asked many times.

Mr. Gammon,

I guessed you were reading at 115200 and debugging at 9600 and that is exactly what you were doing!

Precisely. What is the significance of the baud rates between the serial1 port and serial terminal (debugging)? I didn't have an issue of overrunnig the Serial1 buffer. I've only sent 5 to 18 bytes; furthermore, I initially reported availability() to debugger.

So when you said "Of course I've paired same baud rate" - that was just plain wrong. I asked if you were reading at one speed and debugging at another. That was the question. Of course you "pair the baud rate" between devices. But that wasn't the question.

Sorry, I guess I misunderstood your question. I thought you were referring device to device baud rate

This is very frustrating when you don't read what we are suggesting and don't post your code until asked many times.

As I said before, I was expecting a conceptual quick and simple explanation for the anomaly. I was seeking whether anyone had experienced this phenomenon. My original inquiry was why at 115200 baud every byte comes in okay except value 0x00. "coding badly" posts the error rate at different bauds. My question is why is the error prone to 0x00 and not any other bytes?

Thanks for the discussion. My apology if I was precise enough. Just a "newbie" for now...

My original inquiry was why at 115200 baud every byte comes in okay except value 0x00.

I don't want to nitpick with you, but no, it wasn't. Your initial post was:

hanlee:
I have a device that sends stream of data. Data includes stream of bytes ranging from 0 to 255. When stream of data contains zero (0x00), the available() does not account for it. For example, if the device sends five bytes (0xAA 0x00 0x4B 0x5F 0x36), then the available() returns only 4 instead of 5. Similarly, if another stream contains (0xAA 0x00 0x4B 0x00 0x36), then the available() returns only 3 instead of 5. How can I or What can I do to have available() account for 0x00s?
Thanks

There was no mention of 115200 baud until reply #9.

What is the significance of the baud rates between the serial1 port and serial terminal (debugging)?

The significance is, that the output to serial ports (at least with HardwareSerial) is "blocking". So if you write to a slow port, like at 9600 baud, you are stopping other things from happening. Having said that, I think that the reading is still done with interrupts, so that probably wasn't the overall reason.

I think the overall reason is that you are running at 8 MHz.

I am pleased the problem is solved, and to give you credit, you correctly deduced somewhat earlier that a lower baud rate would help.


My question is why is the error prone to 0x00 and not any other bytes?

To answer this, I guess from the datasheet, and the code, that to achieve the "fastest" baud rates the library uses "Double Speed Operation". A note in the datasheet goes as follows:

The transfer rate can be doubled by setting the U2Xn bit in UCSRnA. Setting this bit only has effect for the asynchronous operation. Set this bit to zero when using synchronous operation.

Setting this bit will reduce the divisor of the baud rate divider from 16 to 8, effectively doubling the transfer rate for asynchronous communication. Note however that the Receiver will in this case only use half the number of samples (reduced from 16 to 8 ) for data sampling and clock recovery, and therefore a more accurate baud rate setting and system clock are required when this mode is used.

There is a bit of an implication there that at high speeds the behaviour is a bit more unreliable. As to why, exactly, it affects 0x00 and not other bytes, I am not sure. However once you get into "unreliable" operation, the unreliability may manifest itself in strange ways.


You raise an interesting point, and reading the datasheet a bit more (around page 191) it appears that there is an internal state machine that attempts to compensate for clock drift called "Asynchronous Data Recovery". Without getting bogged down in too much detail, I am guessing that a 0x00 byte is probably the hardest for the recovery hardware to handle, as it consists of a start bit (0) followed by 8 data bits (all 0). So, nine consecutive zeroes don't give the hardware any "state changes" to "latch onto" to resynchronize. Even a single 1 bit in the middle could help the hardware detect that the clock speed is drifting, and compensate.

I didn't know it did that, and your reported problem has helped increase my understanding of the hardware. Thanks for that!

Mr. Gammon,

I didn't know it did that, and your reported problem has helped increase my understanding of the hardware. Thanks for that!

No, thank you for the in depth analysis of my issue. Your time and information is greatly appreciated. Thanks again.