Go Down

Topic: Serial synchronization? (Read 3401 times) previous topic - next topic

DianneB

My project involves an Arduino Uno in a remoter location connected to sensors (a 10DOF) that sends a group of data of a fixed length every 5 seconds at 300 BAUD.

The serial is via "dumb radio" (no error detection/correction) and is subject to noise and signal drop-outs so bits/bytes may be missing or there may be extra bits as a result of noise.

The serial is received by a Mega and the data group is processed to drive a number of parameter displays (servos).

The problem is that when the serial is interrupted or glitched by noise, the receiver goes out of sync with the transmitter and the received bytes are no longer valid.

I have experimented unsuccessfully with different ideas to restore sync without success.

Receiver code:


Code: [Select]

  struct DOFDATA
{
  uint16_t compass;
  uint16_t pressure;
  uint16_t pitch;
} ;
  static byte counter = 0;
  static byte rxBuffer[sizeof(DOFDATA)];

  DOFDATA dd;
  if (Serial2.available() > 0 && counter < sizeof(rxBuffer))
  {
    rxBuffer[counter++] = Serial2.read();
  }
  // if data complete
  if (counter == sizeof(rxBuffer))
  {
    // make a copy of the received data
    memcpy(&dd, rxBuffer, sizeof(DOFDATA));

    // reset counter for next receive
    counter = 0;

    compass = (dd.compass);
    pressure = (dd.pressure) ;
    pitch = (dd.pitch) ;


Is there a compact and reliable method to determine if the data is good or to 'reset' the receiver. (It isn't critical is one or more received packets are missed.)

Thanks all!
New to Arduino and C++ but retired designer  - happily retired and still playing!

Robin2

#1
Jan 02, 2017, 11:40 am Last Edit: Jan 02, 2017, 11:42 am by Robin2
You have not posted a complete program.

I suspect you need start- and end-markers for your data stream so the Arduno does not get mixed up. have a look at the 3rd example in Serial Input Basics

It would probably also be wise to include some form of checksum so that the validity of the message could be checked. An extra byte containing the XOR of all the other bytes may be sufficient.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

sterretje

#2
Jan 02, 2017, 12:06 pm Last Edit: Jan 02, 2017, 12:07 pm by sterretje
You should not post a snippet like you do; there is nothing for us to relate to. At least post the complete function and preferably post the full code. As this code has my signature all over it ( ;) )

To sync, you might want to add two fields to the struct (that makes it easy to send).
Code: [Select]

struct DOFDATA
{
  byte startmarker;
  uint16_t compass;
  uint16_t pressure;
  uint16_t pitch;
  byte endmarker;
} ;

The startmarker can e.g. be 0x02 and the endmarker can e.g. be 0x03.

Your receiver ignores received data till it receives a startmarker. It next will read the required number of bytes. If the last byte is not the endmarker, you are out-of-sync and the data needs to be discarded (set counter to 0).

If there are values that can not possibly occur in the actual data (e.g. negative values), you can use those for the startmarker and endmarker (use uint16_t in that case instead of bytes).


You need to add a timeout mechanism as well. If the first byte that you detect is 0x02, it's either the startmarker or a data byte (you don't know yet). Let's assume it's part of pitch; in that case you will only read three bytes max (one or two bytes for pitch and the endmarker) and wait 'forever' for the rest.

You know in this case that you will receive 8 bytes; everytime you receive a byte, you set the start 'time' for the timeout counting so you can compare it against the current 'time' (millis()) in a next iteration.

The timeout part could look like
Code: [Select]
...
...
DOFDATA dd;

// time last byte was received
static unsigned long timeoutStarttime = 0;

// if a timeout start time is set
if (timeoutStarttime != 0)
{
  // check timeout
  if (millis() - timeoutStarttime >= 100)
  {
    // discard the data
    counter = 0;
  }
}


if (Serial2.available() > 0 && counter < sizeof(rxBuffer))
{
  rxBuffer[counter++] = Serial2.read();
  // set new start time for timeout
  timeoutStarttime = millis();
}

// if data complete
if (counter == sizeof(rxBuffer))
{
  // reset timeout starttime to so we don't get false timeouts
  timeoutStarttime = 0;

  ...
  ...

Based on 300 baud, you can received 30 bytes per second; so roughly 30ms per byte. the timeout value is set to 100 to play it safe; you can play with the value.

If you understand an example, use it.
If you don't understand an example, don't use it.

Electronics engineer by trade, software engineer by profession. Trying to get back into electronics after 15 years absence.

DianneB

... this code has my signature all over it ( ;)
It should - you posted it for me a few weeks ago ;)

I know it is probably not appreciated to post snippets of code but the  whole program is getting rather lengthy and 95% of the code has nothing to do with the serial.

I have added a checksum and can identify when the Tx and Rx are out of sync or bytes have been changed but I need a way to figure out how to get them back in sync. I will try the start-byte/end-byte idea.

I suppose, if the checksum is invalid, I can just throw away bytes until the next end-byte and then start over.
New to Arduino and C++ but retired designer  - happily retired and still playing!

Robin2

I suppose, if the checksum is invalid, I can just throw away bytes until the next end-byte and then start over.
Yes.

Or maybe you could send a request for a re-transmission.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

DianneB

#5
Jan 02, 2017, 02:53 pm Last Edit: Jan 02, 2017, 03:13 pm by DianneB
Yes.

Or maybe you could send a request for a re-transmission.

...R
The serial is one-way, but the transmission will repeat every 5 seconds anyway.

I have included an "end of field" value at the end of each transmission but that isn't going to cut it! The problem is getting the receive serial register back in sync with the transmit register.

Since the serial glitch may result in MORE THAN or LESS THAN 8 bits received, the receive register will have leftover bits in it so it wont write to the receive buffer until it gets more bits so the byte transfer to the receive buffer will be garbage and will remain screwed up forever after.

... trying to figure out how to clear the receive register if there is a checksum error ......

****************

I ended up putting in
Code: [Select]

   if (checksum != ((compass)+(pressure)+(pitch))) {   
      Serial.println ("Bad checksum") ;
      Serial2.end() ;
      delay (1000) ;
      Serial2.begin(600);
   } 


and that brings it  back in sync - just have to figure out how to  ignore the garbage in the Rx buffer until the next valid receive .....
New to Arduino and C++ but retired designer  - happily retired and still playing!

Robin2

... trying to figure out how to clear the receive register if there is a checksum error ......
Post the latest version of your program.

Maybe all that is needed is a succession of Serial.read()s until Serial.available() returns 0.

You say you have an "end of field" character - but have you also got a start character. The code in the 3rd example Serial Input Basics just ignores everything until it detects a start character.

Another thought is that you may need a timeout - if the gap between characters exceeds some reasonable value then abort the attempt to collect that message.


...R
Two or three hours spent thinking and reading documentation solves most programming problems.

DianneB

Post the latest version of your program.

Maybe all that is needed is a succession of Serial.read()s until Serial.available() returns 0.
I tried that but the stray leftover bits keep the incoming bytes erroneous.

Quote
You say you have an "end of field" character - but have you also got a start character. The code in the 3rd example Serial Input Basics just ignores everything until it detects a start character.
I took the "start byte" and "stop byte" out because they don't accomplish anything and, once glitched, the proper bytes are never received.

When the processor's receive register has "leftover bits", NONE of the received data will ever be valid until the extraneous bits are cleared, because there are less than 8 bits in the receive register.

I was able to get things back in sync by executing the following when the Checksum byte received doesn't match the Checksum calculated in the receiver:

Code: [Select]
 if (checksum != compass + pressure + pitch) {;
    Serial.println ("bad data") ;
    Serial.println (" ") ;
    
  (dataValidflag = 1) ;
  Serial2.end() ;
  delay (500) ;
  Serial2.begin (600) ;
  Serial.println ("reboot receiver") ;
 }  


Thanks everybody for your thoughts - it helped me figure out what was going on!
New to Arduino and C++ but retired designer  - happily retired and still playing!

jremington

I've found it easiest to use a state machine to decode serial transmissions.

States could include "searching for start", "reading message", "end detected", "checksum failure".

DianneB

I've found it easiest to use a state machine to decode serial transmissions.

States could include "searching for start", "reading message", "end detected", "checksum failure".
I suppose that might work if you could shift out one bit at a time and check the next 16 bits for a start character. If you don't get a  start character, continue shifting out out bits one at a time until you do.
New to Arduino and C++ but retired designer  - happily retired and still playing!

Robin2

I suppose that might work if you could shift out one bit at a time and check the next 16 bits for a start character. If you don't get a  start character, continue shifting out out bits one at a time until you do.
That is pretty much what the Arduino USART does.

I am somewhat confused by the "solution" you posted in Reply #7. I don't understand why it is necessary to stop and restart the USART. Is the problem that it starts receiving a byte but the thing never gets completed because no more bit transitions happen? I have never experienced that and I find it hard to imagine that it does happen unless the transmission is completely cut off and not restarted.

At the same time, your solution seems to depend on getting a complete message so that the checksum can be tested - which implies that sufficient characters are received, even if they are the wrong ones.

And I don't see the sense of this
Quote
I took the "start byte" and "stop byte" out because they don't accomplish anything and, once glitched, the proper bytes are never received.
but you have not posted the program in which you tried that system. I have found the concept perfectly reliable - especially when recovering from glitches.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

jremington

The UART sets a "framing error" flag if the bits get out of synchronization, especially if the wrong value for the stop bit is encountered.

There is no need to start and stop the UART or reboot the processor. Just look at the incoming characters, ignoring those that cause error flags to be set.

A valid "sentence" will have a valid start and stop character (as defined by you), a valid checksum, with no error bits set at any time.

DianneB

I don't understand why it is necessary to stop and restart the USART. Is the problem that it starts receiving a byte but the thing never gets completed because no more bit transitions happen? I have never experienced that and I find it hard to imagine that it does happen unless the transmission is completely cut off and not restarted.
Yes, it IS caused by a break in the serial link or noise on the serial line.

Since the serial transfers data one bit at a time, it takes all 8 bits to make a valid byte (plus whatever start and stop bits the UART requires).

Assuming that 4 data bits have been received when the serial link is broken, or maybe a 5th bits is caused by noise so there  are  4 (or 5) bits in the serial receive shift register so the register doesn't flag a full byte being received. Nothing will happen until more bits are received.

When the serial link is restored, the next 3 or 4 bits coming in will be used to complete the current byte in the receive register and that byte will be flagged as complete, even though it is wrong, and the next 3 or 4 bits will be left in the serial shift register. This state will continue until/unless the extra bits in the receive register are cleared and only after the extraneous bits are cleared will the transmitter and receiver be in sync again. Since I don't have the ability to directly "flush" the receiver shift register, the only way I found to clear the "leftover" bits is to disable and re-enable the Serial port.


Quote
At the same time, your solution seems to depend on getting a complete message so that the checksum can be tested - which implies that sufficient characters are received, even if they are the wrong ones.
That is correct. Eventually more bits will be received and the whole message will complete. Comparing the checksum (or CRC, or whatever) received from the transmitter to the checksum calculated from  the message itself is the only way to tell that the Tx and the Rx are out of sync.

Quote
And I don't see the sense of this but you have not posted the program in which you tried that system. I have found the concept perfectly reliable - especially when recovering from glitches.
The code is very lengthy and people get upset when I only post the "snippets" of the serial routines.

Which concept is "perfectly reliable"?  Have you deliberately glitched the serial (by shorting or opening the serial line?
New to Arduino and C++ but retired designer  - happily retired and still playing!

mauried

Id start by using better radios, something like these. HC-12.
http://www.banggood.com/HC-12-433-SI4463-Wireless-Serial-Module-Remote-1000M-With-Antenna-p-973522.html

If your dumb radio cannot reliably transmit the mark state,(idle state when no chars are being sent),  then the receiving usart cannot reliably detect the start bit of a character, which is why it can get out of sync with the sending usart.
This cant be easily fixed in software.

DianneB

Id start by using better radios .....
Not practical. The existing radio is 900 MHz, long range (well over 1 mile) and works regardless of rain, vegetation and with an omni-directional antenna (not a directional antenna), which is important since the transmitter is on the surface of the water, can be anywhere and moving in any direction.

Quote
This cant be easily fixed in software.
I don't know about "easily" but it is FIXED, tested, and working fine.
New to Arduino and C++ but retired designer  - happily retired and still playing!

Go Up