Any chance serial drops zeros?

Just curious. I've built a USB-to-TTL circuit which interfaces with an old serial 3d control device. However, I've noticed that the packets I get sent from the device are occasionally short a byte or two, and from examining the output, it looks like the dropped bytes are zeros, probably zeros in a series.

It's extremely hard to verify this, but that looks like what's happening. I've managed to just ignore that by dropping incomplete packets, but I'm trying to make sure there's nothing wrong with my circuit--and see if the problem is in the Arduino libs, my circuit, or the ages-old serial device.

Has there been any documented problem with Arduino's Serial routines (on pins 0 and 1, ie not SoftSerial) dropping characters in any regime? This is basic 9600,8n1 serial. Just not sure what's going on.

Do you have more than one serial device attached to the TX and RX pins?
Are you dropping ASCII zero characters or null characters '\0'?
I could image a serial routine dropping null characters.

(* jcl *)

Assuming you are dropping 0x00 bytes not ASCII "0" bytes, this problem is likely regardless of your hardware. The serial character is sent as:
space, 8 bits, mark that is:
0, 0000000, 1

The line then stays at 1 until the next character. if you send these very close together the mark time is approximately 1 bit. There is no way to determine if the "mark" - 1 is a mark bit or a "1" in the middle of a data byte. If it looks like it is in the middle of a data byte, it will be discarded as an error since there will not be a mark at the end of the word.

Anyway, if you are not using parity bits, then sometimes you can slow down the transmitting machine. Older systems would sometimes let you specify an "intra character delay" (or something of the sort). What you need is to make sure you never get characters closer together than 10 or 11, bit times, that way the start of the next character is always clear. Parity check or a packet length/parity check would be best.

I have had this trouble sending from a PIC to my PC if I set characters at maximum speed with no delays. I was sending two repeating characters at maximum speed. The PC would sync up in the middle of the character and display two different characters. If I plugged and unplugged the cable, eventually it would start on the right bit boundary and display properly.

Yeah, drspectro pretty well described the problem. I'm posting only as a bit of trivia. This is a known problem and telco companies have fixed it on their lines. Go here to find out all you never wanted to know about how to fix the problem if you control both ends of the line:

That document covers means needed to prevent losing sync timing in synchronous communications channels where a clock is recovered from the modulation stream.

Asynchronous communications does not utilize embedded clock timing but rather relies on both the sender and receiver circuits to use external timing references within specifications for the standard baud rates utilized.

The bit timing is restarted with each new character's start bit. Perhaps you have a case of the receiver and transmitters are mismatched in timing just enough to be sensitive to a all 0 character? You might try setting the baud rate on each channel to a lower speed and see if the symptom still remains.

Lefty

What async speed are you operating at? There are some common ones (115.2 kbps) that are not possibles to get very close to given the Arduino clock rate. Well matched async communictions shouldn't have any problems with null bytes, even if they're repeated, and certainly not with an 'occasional' null...

I agree that trying other baud rates is a good idea (and the most likely issue) The AVR data sheet is pretty readable. andshows 9600 and 4800 at .2% error but 2400 at -.1%,
and 115.2K at -3.5%....

--- nit picking discussion of serial timing follows
That said, repeated Nulls really is a degenerate case. Even if your baud clocks are perfectly matched if the delay between characters is close to one bit time (maybe even up to 11 bit times) the pattern really is ambiguous. As I recall the UART doesnt control that delay either, its just that software usually takes a few clock cycles between character writes.

In any case this should cause an error on the UART because the stop bit wont be in the right place (there wouldnt be a stop bit). The AVR datasheet shows the UART error codes in the UCSRnA. register, but it looks like it gets cleared with each character read. You might be able to look in the Arduino source and see if they check for errors in that register in the serial routines. Or it might be interesting to grab the UART interrupt and service it in your own code that way you could check for frame errors.

If you have time to execute code betweek incoming characters you might be able to read the UCSRnA register yourself, but you would have to read it before the next char comes in.

A serial debug tool might be handy if someone would code it...allow all the serial parameters to be set. There is good autobaud code out there, how about something to identify unknown serial speeds......

Even if your baud clocks are perfectly matched if the delay between characters is close to one bit time (maybe even up to 11 bit times) the pattern really is ambiguous.

No, it's really not. The reception of a byte begins with the falling edge of the "start bit" from the idle state. The internal logic then skips the rest if the start bit, reads the appropriate number of data and parity bits into a shift registers and it's DONE. The one-bit time of "stop-bit" that you're worried about doesn't even need to be checked for validity, and can probably be SHORTER than a full bit length, just so long as the internal logic has time to see the next edge transition. Most uarts sample the bitstreams at 16x the bit clock...

Of course, this does mean that you have to reliably notice the "idle state" to start with, and an "idle" time of greater than one byte time should pretty much guarantee that, but the all nulls bit pattern isn't any worse than many other character patterns (historically a string of capital U's is particularly tough. IIRC, it's a square wave...) Trying to tap into an existing continuous async data stream is pretty tough; you CAN'T reliably detect the stop/start bit combination without having an idle state exceeding one byte time. But zeros are no worse a problem than other patterns (and perhaps less, since there is only one bit that could possibly be the stop bit...) (On transmit, a uart includes the stop bit(s) as part of the transmit shift register, so there should never be a "too short" stop bit.)

(You can of course imagine a BROKEN UART, or more likely a broken implementation of a SW bit-banged UART, that doesn't transmit a proper stop bit, but that would be merely BROKEN, not a degenerate case of async communications behavior.)

(Consider old-fashioned protocols like X/Y/ZModem that happily transmitted binary files all around the pre-internet...)

All that said, I don't know that it helps the original poster any.

Actually, all that did help me, because at least I feel better about not having possibly mangled my circuit or software. I don't have much choice in the communications speed (it's gotta be 96008N1), so it looks like it's just a problem I'll have to deal with.

I am curious that the computer doesn't seem to have as much trouble, but it could be either tighter timing or some unknown cleverness. The big takeaway from this is "no, that's a problem, so your current plan of just dropping suspect packets is probably best, and you can go back to blowing up zombies with your $300 CAD device you glommed from ebay for $20 and ran through your Arduino." Because really, if you can't use homemade electronics to blow up zombies, who IS going to do it?

Seriously, though, thanks--that put my mind at ease.

Point granted on the stop bit timing.

I just went through wiring_serial.c and it doesnt seem to check the error bits. I am looking at whether I can easily chain to it and check for error codes at each character. That would increase the odds of a buffer overflow so you wouldnt want to leave it turned on, but it would be good for testing this type of problem.

If you did have bad or mismatched hardware, the framing errors should pop right out.

Hey vputz
I got a test routine that will report errors on the UART running. First cut is NOT pretty - requires editing the wiring serial file.
However it is only about 10 lines of code.

There HAS to be a better way to link this, but as a quick test it seems to work. I have tested by grounding the RX line while dumping data from Hyperterminal and it definitely throws an error.....

No time tonight to upload, but I will get this posted tomorrow. I am also going to try to figure out a clean way to do this as a separate library rather than editing wiring.

Kluge to check for serial receive errors -- seems to work on my bench top....

the status byte is ORed with the new status after every each character is received. When you check the status after receiving a number of bytes, if the error bits are not set, then there were no errors. If there were errors at any time since the previous status check the error bits will be set. This is much faster than checking for errors after every byte and shouldnt throw off buffering too much.
The routine to read the status register resets the variable to 0 each time it is called.

Finally -- only bits 2-4 indicate errors. I mask them in the test program.


NOTE this involves editing the wiring serial library. Not something you want to do for your production code. I dont know the "official" warnings on this, but be sure you make backup copies so you can put the original files back.

I am still researching the "correct" way to do this, and this shouldnt cause any problems as long as you put the original files back.


Part 1 - Edit wiring_serial.c

// this stuff is near the top of the files
int rx_buffer_head = 0;
int rx_buffer_tail = 0;
//  ********begining of first change*******
// new status variable here
byte status_reg = 0;

//function to return status byte
//status variable reset to 0 with each call
int serialStatus(void)
{
  int temp;
  temp = status_reg;
  status_reg = 0;
  return temp;
}
//  ******end of  first change*********
void beginSerial(long baud)

//**** second change ***********
//  skip further down the file to make this change
//  modify the interrupt/signal routine
// **************************** 
#if defined(__AVR_ATmega168__)
SIGNAL(SIG_USART_RECV)
#else
SIGNAL(SIG_UART_RECV)
#endif
{
#if defined(__AVR_ATmega168__)
      unsigned char c = UDR0;
#else
      unsigned char c = UDR;
#endif
// ************ begin second change *********
//get status byte
               status_reg = status_reg | UCSR0A;
//************  end second change  **********
      int i = (rx_buffer_head + 1) % RX_BUFFER_SIZE;

      // if we should be storing the received character into the location
      // just before the tail (meaning that the head would advance to the
      // current location of the tail), we're about to overflow the buffer
      // and so we don't write the character or advance the head.
      if (i != rx_buffer_tail) {
            rx_buffer[rx_buffer_head] = c;
            rx_buffer_head = i;
      }
}

Part 2 -- edit wiring.h

// you should problably put this near the other serial declarations
int serialStatus(void);

Part 3 -- Demo program

// reads a serial character then echos it back
// checks the status byte before each read
// you really should only check status at the end of a packet
// the delays are probably not necessary
int read_char;
int status;
void setup() 
{ 
  Serial.begin(9600); 
  
  // prints title with ending line break 
  Serial.println("Serial Test"); 
 
  // wait for the long string to be sent 
  delay(100); 
  
 // pinMode(ledPin, OUTPUT);
  
} 


void loop() 
{ 
  if (Serial.available() > 0) 
  { status = serialStatus();
// only bits 2-4 indicate errors  mask the others with 0x1C
// bit 2 = parity
// bit 3 = data overrun
// bit 4 = frame error
    status = status & 0x1C;
    read_char = Serial.read();
    Serial.print("*");
    Serial.print(read_char, BYTE);
    delay(100);
  }
 else
   {delay(25);} 
 
 if (status != 0)
  {Serial.print("#");
   Serial.print(status);
   Serial.print("#");
  }
}

Nice work, drspectro--if I have time I'll give it a run this weekend. The app is for more general deployment so I'll probably stick with what I've got (ie check for errors at the packet level rather than byte level) but this is a good addition to the toolkit.

Thanks vputz
If you get a chance to try this for a test I would be really curious what you find out. I tested with a dead short, so it will be interesting to see how a more realistic failure looks.

I posted in the software section for how to code this "legally"; or if it would be a candidate for a core change. I think the best "legal" way to do this is to essentialy copy the wiring_serial library to a new name say "test_serial" with new function names. Then as long as you only used functions from the new library, the orignial library should not link.

Only problem is, I cant figure out how to make it compile when I do the rename.