My project involves an Arduino Uno in a remoter location connected to sensors (a 10DOF) that sends a group of data of a fixed length every 5 seconds at 300 BAUD.
The serial is via "dumb radio" (no error detection/correction) and is subject to noise and signal drop-outs so bits/bytes may be missing or there may be extra bits as a result of noise.
The serial is received by a Mega and the data group is processed to drive a number of parameter displays (servos).
The problem is that when the serial is interrupted or glitched by noise, the receiver goes out of sync with the transmitter and the received bytes are no longer valid.
I have experimented unsuccessfully with different ideas to restore sync without success.
Receiver code:
struct DOFDATA
{
uint16_t compass;
uint16_t pressure;
uint16_t pitch;
} ;
static byte counter = 0;
static byte rxBuffer[sizeof(DOFDATA)];
DOFDATA dd;
if (Serial2.available() > 0 && counter < sizeof(rxBuffer))
{
rxBuffer[counter++] = Serial2.read();
}
// if data complete
if (counter == sizeof(rxBuffer))
{
// make a copy of the received data
memcpy(&dd, rxBuffer, sizeof(DOFDATA));
// reset counter for next receive
counter = 0;
compass = (dd.compass);
pressure = (dd.pressure) ;
pitch = (dd.pitch) ;
Is there a compact and reliable method to determine if the data is good or to 'reset' the receiver. (It isn't critical is one or more received packets are missed.)
I suspect you need start- and end-markers for your data stream so the Arduno does not get mixed up. have a look at the 3rd example in Serial Input Basics
It would probably also be wise to include some form of checksum so that the validity of the message could be checked. An extra byte containing the XOR of all the other bytes may be sufficient.
You should not post a snippet like you do; there is nothing for us to relate to. At least post the complete function and preferably post the full code. As this code has my signature all over it ( )
To sync, you might want to add two fields to the struct (that makes it easy to send).
The startmarker can e.g. be 0x02 and the endmarker can e.g. be 0x03.
Your receiver ignores received data till it receives a startmarker. It next will read the required number of bytes. If the last byte is not the endmarker, you are out-of-sync and the data needs to be discarded (set counter to 0).
If there are values that can not possibly occur in the actual data (e.g. negative values), you can use those for the startmarker and endmarker (use uint16_t in that case instead of bytes).
You need to add a timeout mechanism as well. If the first byte that you detect is 0x02, it's either the startmarker or a data byte (you don't know yet). Let's assume it's part of pitch; in that case you will only read three bytes max (one or two bytes for pitch and the endmarker) and wait 'forever' for the rest.
You know in this case that you will receive 8 bytes; everytime you receive a byte, you set the start 'time' for the timeout counting so you can compare it against the current 'time' (millis()) in a next iteration.
The timeout part could look like
...
...
DOFDATA dd;
// time last byte was received
static unsigned long timeoutStarttime = 0;
// if a timeout start time is set
if (timeoutStarttime != 0)
{
// check timeout
if (millis() - timeoutStarttime >= 100)
{
// discard the data
counter = 0;
}
}
if (Serial2.available() > 0 && counter < sizeof(rxBuffer))
{
rxBuffer[counter++] = Serial2.read();
// set new start time for timeout
timeoutStarttime = millis();
}
// if data complete
if (counter == sizeof(rxBuffer))
{
// reset timeout starttime to so we don't get false timeouts
timeoutStarttime = 0;
...
...
Based on 300 baud, you can received 30 bytes per second; so roughly 30ms per byte. the timeout value is set to 100 to play it safe; you can play with the value.
sterretje:
... this code has my signature all over it (
It should - you posted it for me a few weeks ago
I know it is probably not appreciated to post snippets of code but the whole program is getting rather lengthy and 95% of the code has nothing to do with the serial.
I have added a checksum and can identify when the Tx and Rx are out of sync or bytes have been changed but I need a way to figure out how to get them back in sync. I will try the start-byte/end-byte idea.
I suppose, if the checksum is invalid, I can just throw away bytes until the next end-byte and then start over.
Or maybe you could send a request for a re-transmission.
...R
The serial is one-way, but the transmission will repeat every 5 seconds anyway.
I have included an "end of field" value at the end of each transmission but that isn't going to cut it! The problem is getting the receive serial register back in sync with the transmit register.
Since the serial glitch may result in MORE THAN or LESS THAN 8 bits received, the receive register will have leftover bits in it so it wont write to the receive buffer until it gets more bits so the byte transfer to the receive buffer will be garbage and will remain screwed up forever after.
... trying to figure out how to clear the receive register if there is a checksum error ......
DianneB:
... trying to figure out how to clear the receive register if there is a checksum error ......
Post the latest version of your program.
Maybe all that is needed is a succession of Serial.read()s until Serial.available() returns 0.
You say you have an "end of field" character - but have you also got a start character. The code in the 3rd example Serial Input Basics just ignores everything until it detects a start character.
Another thought is that you may need a timeout - if the gap between characters exceeds some reasonable value then abort the attempt to collect that message.
Maybe all that is needed is a succession of Serial.read()s until Serial.available() returns 0.
I tried that but the stray leftover bits keep the incoming bytes erroneous.
You say you have an "end of field" character - but have you also got a start character. The code in the 3rd example Serial Input Basics just ignores everything until it detects a start character.
I took the "start byte" and "stop byte" out because they don't accomplish anything and, once glitched, the proper bytes are never received.
When the processor's receive register has "leftover bits", NONE of the received data will ever be valid until the extraneous bits are cleared, because there are less than 8 bits in the receive register.
I was able to get things back in sync by executing the following when the Checksum byte received doesn't match the Checksum calculated in the receiver:
jremington:
I've found it easiest to use a state machine to decode serial transmissions.
States could include "searching for start", "reading message", "end detected", "checksum failure".
I suppose that might work if you could shift out one bit at a time and check the next 16 bits for a start character. If you don't get a start character, continue shifting out out bits one at a time until you do.
DianneB:
I suppose that might work if you could shift out one bit at a time and check the next 16 bits for a start character. If you don't get a start character, continue shifting out out bits one at a time until you do.
That is pretty much what the Arduino USART does.
I am somewhat confused by the "solution" you posted in Reply #7. I don't understand why it is necessary to stop and restart the USART. Is the problem that it starts receiving a byte but the thing never gets completed because no more bit transitions happen? I have never experienced that and I find it hard to imagine that it does happen unless the transmission is completely cut off and not restarted.
At the same time, your solution seems to depend on getting a complete message so that the checksum can be tested - which implies that sufficient characters are received, even if they are the wrong ones.
And I don't see the sense of this
I took the "start byte" and "stop byte" out because they don't accomplish anything and, once glitched, the proper bytes are never received.
but you have not posted the program in which you tried that system. I have found the concept perfectly reliable - especially when recovering from glitches.
The UART sets a "framing error" flag if the bits get out of synchronization, especially if the wrong value for the stop bit is encountered.
There is no need to start and stop the UART or reboot the processor. Just look at the incoming characters, ignoring those that cause error flags to be set.
A valid "sentence" will have a valid start and stop character (as defined by you), a valid checksum, with no error bits set at any time.
Robin2:
I don't understand why it is necessary to stop and restart the USART. Is the problem that it starts receiving a byte but the thing never gets completed because no more bit transitions happen? I have never experienced that and I find it hard to imagine that it does happen unless the transmission is completely cut off and not restarted.
Yes, it IS caused by a break in the serial link or noise on the serial line.
Since the serial transfers data one bit at a time, it takes all 8 bits to make a valid byte (plus whatever start and stop bits the UART requires).
Assuming that 4 data bits have been received when the serial link is broken, or maybe a 5th bits is caused by noise so there are 4 (or 5) bits in the serial receive shift register so the register doesn't flag a full byte being received. Nothing will happen until more bits are received.
When the serial link is restored, the next 3 or 4 bits coming in will be used to complete the current byte in the receive register and that byte will be flagged as complete, even though it is wrong, and the next 3 or 4 bits will be left in the serial shift register. This state will continue until/unless the extra bits in the receive register are cleared and only after the extraneous bits are cleared will the transmitter and receiver be in sync again. Since I don't have the ability to directly "flush" the receiver shift register, the only way I found to clear the "leftover" bits is to disable and re-enable the Serial port.
At the same time, your solution seems to depend on getting a complete message so that the checksum can be tested - which implies that sufficient characters are received, even if they are the wrong ones.
That is correct. Eventually more bits will be received and the whole message will complete. Comparing the checksum (or CRC, or whatever) received from the transmitter to the checksum calculated from the message itself is the only way to tell that the Tx and the Rx are out of sync.
And I don't see the sense of this but you have not posted the program in which you tried that system. I have found the concept perfectly reliable - especially when recovering from glitches.
The code is very lengthy and people get upset when I only post the "snippets" of the serial routines.
Which concept is "perfectly reliable"? Have you deliberately glitched the serial (by shorting or opening the serial line?
Id start by using better radios, something like these. HC-12.
If your dumb radio cannot reliably transmit the mark state,(idle state when no chars are being sent), then the receiving usart cannot reliably detect the start bit of a character, which is why it can get out of sync with the sending usart.
This cant be easily fixed in software.
Not practical. The existing radio is 900 MHz, long range (well over 1 mile) and works regardless of rain, vegetation and with an omni-directional antenna (not a directional antenna), which is important since the transmitter is on the surface of the water, can be anywhere and moving in any direction.
This cant be easily fixed in software.
I don't know about "easily" but it is FIXED, tested, and working fine.
If you have 45 years experience in communications, then your 900 MHz link is or should be FM, which will not be susceptible to noise. Do you know for sure you are not getting interference from other 900MHz users?
Are you checking the received status of each byte received by the USART?
You may also need to give each byte a parity check bit and the USART can check this, also.
Paul_KD7HB:
... your 900 MHz link is or should be FM, which will not be susceptible to noise.[/quote/]
I am not using the radios yet - I am developing the "end points" in a hardwired configuration and providing the serial lines with as much immunity as possible. The transmit unit, in particular, is in an electrically noisy environment.
Do you know for sure you are not getting interference from other 900MHz users?
I have been using the same radios for about 12 years for a video link and interference is fairly uncommon.
Are you checking the received status of each byte received by the USART? You may also need to give each byte a parity check bit and the USART can check this, also.
I don't think that's necessary. The data isn't time-critical so I can tolerate loosing one or two transmissions. The important part is to not 'act' on erroneous data.
As to "if you have 45 years (design) experience" - I wont dignify that with a response.
There is something important that Robin's tutorial did not tell you and it is very important for your program.
Asynchronous communications refers to the byte or character being sent/received. It's starting time is not defined. HOWEVER, on the BIT level, the communication is SYNCHRONOUS. In other words, the UART or software replacement for the uart WILL give you a o bit or a 1 bit for each position of the byte or character based on the CLOCK, not whether there is noise or not on the input line. That is why most asynchronous data uses a parity bit so the UART can determine if the byte created is what was sent.
A properly created check sum can tell you if one byte of the message had a character that has a 1 or 0 in the wrong place. Some check sums can tell you if you had 2 bits wrong, but still had the correct parity.
I think you program is not able to do what you believe it is doing.
Noise and pulling the cable out while a byte is being transferred should not make a difference. The receiving end will just read rubbish while it is reading a byte. A checksum might pick it up or might not pick it up; a CRC will be more solid.
Connecting a cable during a byte transfer (either at the transmitter or the receiver) will indeed result in the described problem of e.g. only receiving the first 4 bits of a byte followed by the first 4 bits of the next byte (all bytes in the packet are basically shifted 4 bits).
Framing with startmarker and endmarker also helps to get it more reliable. And a timeout mechanism will prevent waiting for e.g. the last byte (if you missed the first one) which is actually the first byte of the next packet.
Personally I think that for the most reliable communication all those measures should be implemented.
DianneB:
Assuming that 4 data bits have been received when the serial link is broken, or maybe a 5th bits is caused by noise so there are 4 (or 5) bits in the serial receive shift register so the register doesn't flag a full byte being received. Nothing will happen until more bits are received.
When the serial link is restored, the next 3 or 4 bits coming in will be used to complete the current byte in the receive register and that byte will be flagged as complete, even though it is wrong,
This is exactly why I suggested introducing a timeout when the interval between successive bytes is unreasonably long. That it the point at which the USART should be restarted so that "good" data in the next message is not confused with the remnants of data from the previous failed message.
If the timeout is triggered that can also be a signal that the current message is incomplete.
Using 600baud there should be about 60 characters per second or an interval of about 16700 microsecs between the start of one byte and the start of the next. I suspect a timeout interval of, say, 32000 µsecs would be appropriate. (E&OE)
)