Go Down

Topic: New - Robust Serial Data Transfer Library (Transfers 16-bit values) (Read 688 times) previous topic - next topic

Power_Broker

Hello everybody,

Working with serial data can be a real pain. Many newcomers to the hobby have trouble packaging, sending, and parsing serial data, yet desperately need it for their project to succeed. This is especially true if they want to send 16-bit values.

I've just finished writing a new library that will take care of ALL of that for the user (github link).

Developed originally for Arduino RC aircraft communications, this library was written to ensure that data is quickly sent and received with 100% accuracy. Don't worry, if you're not working on a drone or plane but still use serial data, this library will still work for you ;)

Here are some neat features:
- Transfers 16-bit values
- Packets can have variable length --> only send what ya need
- works with hardware UART and software defined serial ports
- works at all baud rates
- use a start byte, payload length (in bytes), payload, 8-bit checksum of payload, and end byte


Check it out and let me know what you think or if any errors pop up. Thanks!
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Robin2

Two or three hours spent thinking and reading documentation solves most programming problems.

Power_Broker

Is it much different from Serial Input Basics ?

...R
Yes, it is quite different actually.

Although your tutorial works wonderfully and is a very effective learning resource, my library offers a few extra perks.

Firstly, it's a library that the user only needs to download and include. Instead of having to learn and rewrite code in a tutorial (again - great for learning, but not for implementing in a complex project), the user only needs to learn a few specifics on how to interface with the library. Having the code in a library helps readability and limit the size of the main code. It also helps easily reproduce functionality across sketches.

Secondly, the library takes care of dynamic sized data packets (I don't think your tutorial explained how to do that). I also didn't see any info on how to send 16-bit values in your tutorial - which my library handles automatically.

Thirdly, my code uses a payload size and checksum byte instead of delimiters.

Lastly, there are "exception handles" incorporated into my library that gives the parent code (and/or user) a lot more information as to what exactly is happening with the serial data. For instance, if the main code attempts to get a serial packet from the USART buffer, the return value of getData() will tell you one of the following reports:

 - A full packet was successfully parsed
 - No data was present (no packets to parse yet)
 - A packet was received but the payload byte was of incorrect value (possible packet corruption detected)
 - A packet was received but the checksum received was incorrect (possible packet corruption detected)
 - A packet was received but the ending byte was not found (possible packet corruption detected)
 - The starting byte of a packet was found, but the rest of the packet took too long to show up in the buffer (possible packet drop detected - hard timeout defined by user)


Robin2, since I know you have more experience with Arduinos than I have, I invite you to check out the github page and let me know what you think. To be honest, I do have an edit or two I'd like to make soon, but it works quite well as is.
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Robin2

You seem to have described the additional functionality very well.

And you are quite correct, my tutorial is aimed firmly at teaching the basics.

I clicked your Github link but I don't see any documentation for your library - perhaps you can post a link to it.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Power_Broker

Hmmm, I do have a basic readme and a set of example sketches in the Githup repository. I'll go ahead and post the code and the examples with some description.


The readme:
Code: [Select]

AirComms:
Robust Arduino serial data transfer library optimized for speed and accuracy

- Quickly and reliably transfer up to 20 16bit values from one Arduino to another
works with hardwired serial connections or with wireless radios such as XBees or bluetooth modules
- Library works with with hardware and/or software serial ports
- Check out the examples file for details on how to use this library



The library consists of only one header and one cpp file (each):

The header:
Code: [Select]

#include "Arduino.h"




#ifndef AirComms_cpp

#define AirComms_cpp



#define DATA_LEN 20
#define BUFF_LEN DATA_LEN * 3
#define START_BYTE 0x7E //dataframe start byte
#define END_BYTE 0xEF //dataframe end byte

//incoming serial data/parsing errors
#define NO_DATA 0
#define SERIAL_BUFF_ERROR -1
#define END_BYTE_ERROR -2
#define CHECKSUM_ERROR -3
#define TIMEOUT_ERROR -4
#define PAYLOAD_ERROR -5




class AirComms
{
public:
//data received
uint16_t incomingArray[DATA_LEN] = { 0 };

//data to send
uint16_t outgoingArray[DATA_LEN] = { 0 };




//initialize the AirComms class
void begin(Stream& stream);

//change the UART buffer timeout (10ms by default)
void setReceiveTimout(byte timeout);

//send a selection of data from outgoingArray
bool sendData(bool indiciesToSendArray[]);

//update incomingArray with new data if available
int8_t getData();




private:
//serial stream
Stream* _serial;

//data processing buffers
byte inBuff[BUFF_LEN] = { 0 };

//receive timeout of 10ms by default
uint16_t timeout = 10;

//checksum for determining validity of transfer
byte checksum = 0;




//find 8-bit checksum of message
bool findChecksum(byte len);

//find 8 - bit checksum of message
bool findChecksum(bool indiciesArray[]);

//process raw data and stuff into dataArray
void processData(byte payloadLen);
};

#endif



The cpp:
Code: [Select]

#include "AirComms.h"




//initialize the AirComms class
void AirComms::begin(Stream &stream)
{
_serial = &stream;

return;
}



//change the UART buffer timeout (10ms by default)
void AirComms::setReceiveTimout(byte _timeout)
{
timeout = _timeout;

return;
}




//find 8-bit checksum of message
bool AirComms::findChecksum(byte len)
{
//reset checksum
checksum = 0;

//check if len is valid
if (len < BUFF_LEN)
{
//compute checksum
for (byte i = 0; i < len; i++)
{
checksum = checksum + inBuff[i];
}

checksum = (~checksum) + 1;
}
else
{
//couldn't update checksum
return false;
}

//checksum updated
return true;
}




//find 8-bit checksum of message
bool AirComms::findChecksum(bool indiciesArray[])
{
//reset checksum
checksum = 0;

//compute checksum
for (byte i = 0; i < sizeof(indiciesArray); i++)
{
//check if index is valid
if (indiciesArray[i])
{
checksum = checksum + i + (byte)(0xff & (outgoingArray[i] >> 8)) + (byte)(0xff & outgoingArray[i]);
}
else
{
//reset checksum
checksum = 0;

//checksum couldn't be updated
return false;
}
}

checksum = (~checksum) + 1;

return true;
}




//send a selection of data from outgoingArray
bool AirComms::sendData(bool indiciesToSendArray[])
{
//update checksum before sending dataframe
if (!findChecksum(indiciesToSendArray))
{
//couldn't update checksum - return to main code
return false;
}

//send START_BYTE
_serial->write(START_BYTE);

//send payload length in bytes
_serial->write(sizeof(indiciesToSendArray) * 3);

//send payload
for (byte i = 0; i < sizeof(indiciesToSendArray); i++)
{
//test if the data at this index should be sent
if (indiciesToSendArray[i])
{
_serial->write(i); //message ID
_serial->write((byte)(0xff & (outgoingArray[i] >> 8))); //data MSB
_serial->write((byte)(0xff & outgoingArray[i])); //data LSB
}
}

//send checksum
_serial->write(checksum);

//send END_BYTE
_serial->write(END_BYTE);

return true;
}




//update incomingArray with new data if available
int8_t AirComms::getData()
{
uint32_t startTime = 0;
uint32_t endTime = 0;

byte payloadLen = 0;

bool startFound = false;

//see if any data is in the serial buffer
if (_serial->available())
{
//process only what bytes are currently in the buffer when looking for the START_BYTE
while (_serial->available())
{
//check if any bytes in the buffer are
if (_serial->read() == START_BYTE)
{
//start of a dataframe has been found - time to start looking for the data
startFound = true;

//break the "while available" loop
break;
}
}

//determine if the start of frame byte was found
if (startFound)
{
//wait for the payload byte
while (_serial->available() == 0);

//read in the number of bytes in the payload of the packet
payloadLen = _serial->read();

//sanity check for the payload length (should be a multiple of 3 - 1 byte for ID, 2 for raw data)
if ((payloadLen > (DATA_LEN * 3)) || (payloadLen % 3))
{
//oops, bad payload length value
return -5;
}

//prime the timeout timer
startTime = millis();
endTime = millis();

//wait for rest of dataframe to arrive with timeout
while (_serial->available() < (payloadLen + 2)) //add 2 (1 for checksum and 1 for END_BYTE)
{
//update timer
endTime = millis();

//test for timeout
if ((endTime - startTime) >= timeout)
{
//oops, data didn't arrive on time - better get back to processing other things
return -4;
}
}

//stuff all payload bytes in the buffer for processing
for (byte i = 0; i < payloadLen; i++)
{
inBuff[i] = _serial->read();
}

//update checsum before processing
findChecksum(payloadLen);

//test received checksum
if (_serial->read() != checksum)
{
//dang, checksums don't match - can't trust the data - get back to the main code
return -3;
}

//test END_BYTE
if (_serial->read() != END_BYTE)
{
//ugh, END_BYTE wasn't found in the right spot - can't trust the data - get back to the main code
return -2;
}

//process raw data and stuff into dataArray only if all data validity tests are passed
processData(payloadLen);

//nocie, everything checked out
return 1;
}
else
{
//looks like we had garbage bytes in the serial buffer
return -1;
}
}

//no bytes to process
return 0;
}




void AirComms::processData(byte payloadLen)
{
//check if payloadLen is valid
if ((payloadLen <= (DATA_LEN * 3)) && (!(payloadLen % 3)))
{
for (byte i = 0; i < payloadLen; i = i + 3)
{
//sanity check for messageID
if (inBuff[i] <= DATA_LEN)
{
//stuff the 16-bit data (data arrives bigendian)
incomingArray[inBuff[i]] = (inBuff[i + 1] << 8) | inBuff[i + 2];
}
}
}

return;
}
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Power_Broker

This library also has two sketches for the example provided. One sketch is for the transmitter and one is for the receiver.

Note that the user can use both transmitter and receiver functionalities of the library in the same sketch if the proper hardware is used (i.e. user has a full duplex radios or has multiple half duplex radios). For example, My plane has separate half duplex radios on board: one for receiving control surface commands from the hand controller and a separate radio to send telemetry data back to the hand controller.


The TX code:
Code: [Select]

#include <AirComms.h>




//create an instance of the AirComms class
AirComms myRadio;




//array to identify which datafields need to be sent to the other device
bool myArray[DATA_LEN] = {false};




void setup()
{
  //setup power LED
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
 
  //PC debugging port
  Serial.begin(115200);

  //radio port
  Serial1.begin(9600);

  //initialize radio
  myRadio.begin(Serial1);

  //update datafields
  myRadio.outgoingArray[0] = 2000;
  myRadio.outgoingArray[1] = 2001;
  myRadio.outgoingArray[2] = 3000;
  myRadio.outgoingArray[3] = 3001;

  //update array
  myArray[0] = true;
  myArray[1] = true;
  myArray[2] = true;
  myArray[3] = true;
}




void loop()
{
  //send the data
  myRadio.sendData(myArray);

  //wait a little bit
  delay(500);
}






The RX code:
Code: [Select]

#include <AirComms.h>




//create an instance of the AirComms class
AirComms myRadio;




bool myArray[DATA_LEN] = {false};




void setup()
{
  //setup power LED
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
 
  //PC debugging port
  Serial.begin(115200);

  while(!Serial);

  //radio port
  Serial1.begin(9600);

  //initialize radio
  myRadio.begin(Serial1);
}




void loop()
{
  //get new data if available
  int report = myRadio.getData();

  //figure out if data was available - if so, determine if the transfer successful
  if(report == 1)
  {
    Serial.println("Success!");
    Serial.print("\tNew Data: ");
    for(byte i=0; i<DATA_LEN; i++)
    {
      if(i != 0)
      {
        Serial.print(", ");
      }
     
      Serial.print(myRadio.incomingArray[i]);
    }
    Serial.println();
  }
  else if(report == NO_DATA)
  {
    Serial.println("No Data Available");
  }
  else if(report == SERIAL_BUFF_ERROR)
  {
    Serial.println("ERROR - SERIAL_BUFF_ERROR");
  }
  else if(report == END_BYTE_ERROR)
  {
    Serial.println("ERROR - END_BYTE_ERROR");
  }
  else if(report == CHECKSUM_ERROR)
  {
    Serial.println("ERROR - CHECKSUM_ERROR");
  }
  else if(report == TIMEOUT_ERROR)
  {
    Serial.println("ERROR - TIMEOUT_ERROR");
  }
  else if(report == PAYLOAD_ERROR)
  {
    Serial.println("ERROR - PAYLOAD_ERROR");
  }
  else
  {
    Serial.print("ERROR - UNKNOWN ERROR OCCURRED: ");
    Serial.println(report);
  }

  delay(500);
}



"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Power_Broker

This post will describe the main functionality.


The entire library is a basic wrapper for a class. This single class has both transmitting and receiving logic inside of it through the use of different member functions.


The TX Side:
The data structure that holds all data that can be sent to the receiver is an array of 16-bit integers. This array is called "outgoingArray" and has a fixed length of 20. In order to send any of the values in this array, a few things must happen:

1.) Identify which index in the array corresponds to the value to be sent. Insert the new value into the array at that index. Repeat for all values the user desires to send in the next packet.

For instance, on my plane, I might want to constantly send 3 data values: elevator servo angle, aileron servo angle, and rudder servo angle. In order to keep track of which value corresponds to which servo angle, I make a convention. I might say elevator servo angle is always stored in outgoingArray[0], aileron servo angle is always stored in outgoingArray[1], and rudder servo angle is always stored in outgoingArray[2].

Note that "outgoingArray" is a public array and is accessed directly.

2.) Create a boolean array with 20 elements (the size of "outgoingArray" - defined by DATA_LEN). At each index, insert a "True" if you want to send the 16-bit value in the "outgoingArray" at the same index - false where the 16-bit value should not be sent.

For instance, if I only want to send the aileron servo angle for the next transition, I would make the following array:

Code: [Select]

bool outputMask[] = {False};
outputMask[2] = {True};       //aileron data is at index 2 in outgoingArray, all other indices are false and so that no other data will be sent except the aileron data



3.) Pass that array to "sendData()" and watch the desired values show up on the receiving Arduino.


The RX Side:
1.) Periodically call "getData()" to poll the serial buffer and see if data is present for parsing. If data is present, one of 5 reports will be returned, 0 if no data was present. Here is the list of error returns:

Code: [Select]

//incoming serial data/parsing errors
#define NO_DATA 0
#define SERIAL_BUFF_ERROR -1
#define END_BYTE_ERROR -2
#define CHECKSUM_ERROR         -3
#define TIMEOUT_ERROR -4
#define PAYLOAD_ERROR -5


If data was successfully parsed, the function "getData()" will return a 1.

2.) If data was successfully parsed, data in "incomingArray" has been updated. It is up to the user to figure out which indices were changed. It is also up to the user to define which index corresponds to which type of data value - BUT it must remain consistent with which index corresponds to which type of data value in "outgoingArray".

For instance, if the aileron servo angle to be transmitted is always stored in outgoingArray[1], then the aileron servo angle that was received must always stored in incomingArray[1].

Note that "incomingArray" is also a public array and is accessed directly.
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Robin2

Hmmm, I do have a basic readme
This is the only text I see in the ReadMe
Quote
AirComms
Robust Arduino serial data transfer library optimized for speed and accuracy

Quickly and reliably transfer up to 20 16bit values from one Arduino to another
works with hardwired serial connections or with wireless radios such as XBees or bluetooth modules
Library works with with hardware and/or software serial ports
Check out the examples file for details on how to use this library
By documentation I mean something equivalent to this from the AccelStepper library - and it has some serious gaps!

In my experience examples are no substitute for a broad explanation of all the library functions.


...R
Two or three hours spent thinking and reading documentation solves most programming problems.

giova014

A nice library, often I have to write code with serial communication, this makes it easy and fast to implement.

Some points that I find limiting factors:
- Maximum data length of 20
- Data must be uint16_t and assigned manually
What if I want to send a struct containing multiple fields? I must worry about the size and assign manually to outgoingArray / from incomingArray using memcpy, for example.

Do you consider making it more generic, like EEPROM.read() and EEPROM.get(), where you can pass any data type and let the function handle it for you?


Arduino!!

Power_Broker

A nice library, often I have to write code with serial communication, this makes it easy and fast to implement.
Thank you!



Some points that I find limiting factors:
- Maximum data length of 20
- Data must be uint16_t and assigned manually
I absolutely agree. However, the data length can be edited by the user to hold more or fewer elements, but I know that can be difficult for new users and a pain for everyone else. I feel that most users for the vast majority of projects don't need more than 20 data fields.

I also agree with the manual assignment of the data array. While this may be less user friendly, it also provides a lot more control over handling the data.

Again, definitely valid points, but most of the functionality and design of the library was based off my project needs. I needed a quick and flexible way to send 16-bit values with a 100% guarantee that the data received was in fact the data originally sent. Any errors in communication will likely result in my plane falling out of the sky.

If enough people are interested, though, I would be willing to write another library - one that is simpler, easier to use, yet still robust.

All: Please reply below if you are interested in this.
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Power_Broker

I didn't explicitly say this in the intro post, but another cool feature of this library is that there is a timeout in-case only part of a packet arrives and the rest doesn't.
"The desire that guides me in all I do is the desire to harness the forces of nature to the service of mankind."
   - Nikola Tesla

Go Up