New - Robust Serial Data Transfer Library (Transfers 16-bit values)

Community Exhibition / Gallery
1

Hello everybody,

Working with serial data can be a real pain. Many newcomers to the hobby have trouble packaging, sending, and parsing serial data, yet desperately need it for their project to succeed. This is especially true if they want to send 16-bit values.

I've just finished writing a new library that will take care of ALL of that for the user (github link).

Developed originally for Arduino RC aircraft communications, this library was written to ensure that data is quickly sent and received with 100% accuracy. Don't worry, if you're not working on a drone or plane but still use serial data, this library will still work for you :wink:

Here are some neat features:

  • can be downloaded via the Arduino IDE's Libraries Manager (search "SerialTransfer.h")
  • works with "software-serial" libraries
  • is non blocking
  • uses packet delimiters
  • uses consistent overhead byte stuffing
  • uses CRC-8 (Polynomial 0x9B with lookup table)
  • allows the use of dynamically sized packets (packets can have payload lengths anywhere from 1 to 254 bytes)
  • can transfer bytes, ints, floats, and even structs!!

Check it out and let me know what you think or if any errors pop up. Thanks!

2

Is it much different from Serial Input Basics ?

...R

3

Robin2:
Is it much different from Serial Input Basics ?

...R

Yes, it is quite different actually.

Although your tutorial works wonderfully and is a very effective learning resource, my library offers a few extra perks.

Firstly, it's a library that the user only needs to download and include. Instead of having to learn and rewrite code in a tutorial (again - great for learning, but not for implementing in a complex project), the user only needs to learn a few specifics on how to interface with the library. Having the code in a library helps readability and limit the size of the main code. It also helps easily reproduce functionality across sketches.

Secondly, the library takes care of dynamic sized data packets (I don't think your tutorial explained how to do that). I also didn't see any info on how to send 16-bit values in your tutorial - which my library handles automatically.

Thirdly, my code uses a payload size and checksum byte instead of delimiters.

Lastly, there are "exception handles" incorporated into my library that gives the parent code (and/or user) a lot more information as to what exactly is happening with the serial data. For instance, if the main code attempts to get a serial packet from the USART buffer, the return value of getData() will tell you one of the following reports:

  • A full packet was successfully parsed
  • No data was present (no packets to parse yet)
  • A packet was received but the payload byte was of incorrect value (possible packet corruption detected)
  • A packet was received but the checksum received was incorrect (possible packet corruption detected)
  • A packet was received but the ending byte was not found (possible packet corruption detected)
  • The starting byte of a packet was found, but the rest of the packet took too long to show up in the buffer (possible packet drop detected - hard timeout defined by user)

Robin2, since I know you have more experience with Arduinos than I have, I invite you to check out the github page and let me know what you think. To be honest, I do have an edit or two I'd like to make soon, but it works quite well as is.

4

You seem to have described the additional functionality very well.

And you are quite correct, my tutorial is aimed firmly at teaching the basics.

I clicked your Github link but I don't see any documentation for your library - perhaps you can post a link to it.

...R

5

Hmmm, I do have a basic readme and a set of example sketches in the Githup repository. I'll go ahead and post the code and the examples with some description.

The readme:

AirComms:
Robust Arduino serial data transfer library optimized for speed and accuracy

- Quickly and reliably transfer up to 20 16bit values from one Arduino to another
works with hardwired serial connections or with wireless radios such as XBees or bluetooth modules
- Library works with with hardware and/or software serial ports
- Check out the examples file for details on how to use this library

The library consists of only one header and one cpp file (each):

The header:

#include "Arduino.h"




#ifndef AirComms_cpp

#define AirComms_cpp



#define DATA_LEN	20
#define BUFF_LEN	DATA_LEN * 3
#define START_BYTE	0x7E		//dataframe start byte
#define END_BYTE	0xEF		//dataframe end byte

//incoming serial data/parsing errors
#define NO_DATA			0
#define SERIAL_BUFF_ERROR	-1
#define END_BYTE_ERROR		-2
#define CHECKSUM_ERROR		-3
#define TIMEOUT_ERROR		-4
#define PAYLOAD_ERROR		-5




class AirComms
{
public:
	//data received
	uint16_t incomingArray[DATA_LEN] = { 0 };

	//data to send
	uint16_t outgoingArray[DATA_LEN] = { 0 };




	//initialize the AirComms class
	void begin(Stream& stream);

	//change the UART buffer timeout (10ms by default)
	void setReceiveTimout(byte timeout);

	//send a selection of data from outgoingArray
	bool sendData(bool indiciesToSendArray[]);

	//update incomingArray with new data if available
	int8_t getData();




private:
	//serial stream
	Stream* _serial;

	//data processing buffers
	byte inBuff[BUFF_LEN] = { 0 };

	//receive timeout of 10ms by default
	uint16_t timeout = 10;

	//checksum for determining validity of transfer
	byte checksum = 0;




	//find 8-bit checksum of message
	bool findChecksum(byte len);

	//find 8 - bit checksum of message
	bool findChecksum(bool indiciesArray[]);

	//process raw data and stuff into dataArray
	void processData(byte payloadLen);
};

#endif

The cpp:

#include "AirComms.h"




//initialize the AirComms class
void AirComms::begin(Stream &stream)
{
	_serial = &stream;

	return;
}



//change the UART buffer timeout (10ms by default)
void AirComms::setReceiveTimout(byte _timeout)
{
	timeout = _timeout;

	return;
}




//find 8-bit checksum of message
bool AirComms::findChecksum(byte len)
{
	//reset checksum
	checksum = 0;

	//check if len is valid
	if (len < BUFF_LEN)
	{
		//compute checksum
		for (byte i = 0; i < len; i++)
		{
			checksum = checksum + inBuff[i];
		}

		checksum = (~checksum) + 1;
	}
	else
	{
		//couldn't update checksum
		return false;
	}

	//checksum updated
	return true;
}




//find 8-bit checksum of message
bool AirComms::findChecksum(bool indiciesArray[])
{
	//reset checksum
	checksum = 0;

	//compute checksum
	for (byte i = 0; i < sizeof(indiciesArray); i++)
	{
		//check if index is valid
		if (indiciesArray[i])
		{
			checksum = checksum + i + (byte)(0xff & (outgoingArray[i] >> 8)) + (byte)(0xff & outgoingArray[i]);
		}
		else
		{
			//reset checksum
			checksum = 0;

			//checksum couldn't be updated
			return false;
		}
	}

	checksum = (~checksum) + 1;

	return true;
}




//send a selection of data from outgoingArray
bool AirComms::sendData(bool indiciesToSendArray[])
{
	//update checksum before sending dataframe
	if (!findChecksum(indiciesToSendArray))
	{
		//couldn't update checksum - return to main code
		return false;
	}

	//send START_BYTE
	_serial->write(START_BYTE);
	
	//send payload length in bytes
	_serial->write(sizeof(indiciesToSendArray) * 3);

	//send payload
	for (byte i = 0; i < sizeof(indiciesToSendArray); i++)
	{
		//test if the data at this index should be sent
		if (indiciesToSendArray[i])
		{
			_serial->write(i);										//message ID
			_serial->write((byte)(0xff & (outgoingArray[i] >> 8)));	//data MSB
			_serial->write((byte)(0xff & outgoingArray[i]));		//data LSB
		}		
	}

	//send checksum
	_serial->write(checksum);

	//send END_BYTE
	_serial->write(END_BYTE);

	return true;
}




//update incomingArray with new data if available
int8_t AirComms::getData()
{
	uint32_t startTime = 0;
	uint32_t endTime = 0;

	byte payloadLen = 0;

	bool startFound = false;

	//see if any data is in the serial buffer
	if (_serial->available())
	{
		//process only what bytes are currently in the buffer when looking for the START_BYTE
		while (_serial->available())
		{
			//check if any bytes in the buffer are 
			if (_serial->read() == START_BYTE)
			{
				//start of a dataframe has been found - time to start looking for the data
				startFound = true;
				
				//break the "while available" loop
				break;
			}
		}

		//determine if the start of frame byte was found
		if (startFound)
		{
			//wait for the payload byte
			while (_serial->available() == 0);

			//read in the number of bytes in the payload of the packet
			payloadLen = _serial->read();

			//sanity check for the payload length (should be a multiple of 3 - 1 byte for ID, 2 for raw data)
			if ((payloadLen > (DATA_LEN * 3)) || (payloadLen % 3))
			{
				//oops, bad payload length value
				return -5;
			}

			//prime the timeout timer
			startTime = millis();
			endTime = millis();

			//wait for rest of dataframe to arrive with timeout
			while (_serial->available() < (payloadLen + 2)) //add 2 (1 for checksum and 1 for END_BYTE)
			{
				//update timer
				endTime = millis();

				//test for timeout
				if ((endTime - startTime) >= timeout)
				{
					//oops, data didn't arrive on time - better get back to processing other things
					return -4;
				}
			}

			//stuff all payload bytes in the buffer for processing
			for (byte i = 0; i < payloadLen; i++)
			{
				inBuff[i] = _serial->read();
			}

			//update checsum before processing
			findChecksum(payloadLen);

			//test received checksum
			if (_serial->read() != checksum)
			{
				//dang, checksums don't match - can't trust the data - get back to the main code
				return -3;
			}

			//test END_BYTE
			if (_serial->read() != END_BYTE)
			{
				//ugh, END_BYTE wasn't found in the right spot - can't trust the data - get back to the main code
				return -2;
			}

			//process raw data and stuff into dataArray only if all data validity tests are passed
			processData(payloadLen);

			//nocie, everything checked out
			return 1;
		}
		else
		{
			//looks like we had garbage bytes in the serial buffer
			return -1;
		}
	}
	
	//no bytes to process
	return 0;
}




void AirComms::processData(byte payloadLen)
{
	//check if payloadLen is valid
	if ((payloadLen <= (DATA_LEN * 3)) && (!(payloadLen % 3)))
	{
		for (byte i = 0; i < payloadLen; i = i + 3)
		{
			//sanity check for messageID
			if (inBuff[i] <= DATA_LEN)
			{
				//stuff the 16-bit data (data arrives bigendian)
				incomingArray[inBuff[i]] = (inBuff[i + 1] << 8) | inBuff[i + 2];
			}
		}
	}

	return;
}
6

This library also has two sketches for the example provided. One sketch is for the transmitter and one is for the receiver.

Note that the user can use both transmitter and receiver functionalities of the library in the same sketch if the proper hardware is used (i.e. user has a full duplex radios or has multiple half duplex radios). For example, My plane has separate half duplex radios on board: one for receiving control surface commands from the hand controller and a separate radio to send telemetry data back to the hand controller.

The TX code:

#include <AirComms.h>




//create an instance of the AirComms class
AirComms myRadio;




//array to identify which datafields need to be sent to the other device
bool myArray[DATA_LEN] = {false};




void setup()
{
  //setup power LED
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
  
  //PC debugging port
  Serial.begin(115200);

  //radio port
  Serial1.begin(9600);

  //initialize radio
  myRadio.begin(Serial1);

  //update datafields
  myRadio.outgoingArray[0] = 2000;
  myRadio.outgoingArray[1] = 2001;
  myRadio.outgoingArray[2] = 3000;
  myRadio.outgoingArray[3] = 3001;

  //update array
  myArray[0] = true;
  myArray[1] = true;
  myArray[2] = true;
  myArray[3] = true;
}




void loop()
{
  //send the data
  myRadio.sendData(myArray);

  //wait a little bit
  delay(500);
}

The RX code:

#include <AirComms.h>




//create an instance of the AirComms class
AirComms myRadio;




bool myArray[DATA_LEN] = {false};




void setup()
{
  //setup power LED
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
  
  //PC debugging port
  Serial.begin(115200);

  while(!Serial);

  //radio port
  Serial1.begin(9600);

  //initialize radio
  myRadio.begin(Serial1);
}




void loop()
{
  //get new data if available
  int report = myRadio.getData();

  //figure out if data was available - if so, determine if the transfer successful
  if(report == 1)
  {
    Serial.println("Success!");
    Serial.print("\tNew Data: ");
    for(byte i=0; i<DATA_LEN; i++)
    {
      if(i != 0)
      {
        Serial.print(", ");
      }
      
      Serial.print(myRadio.incomingArray[i]);
    }
    Serial.println();
  }
  else if(report == NO_DATA)
  {
    Serial.println("No Data Available");
  }
  else if(report == SERIAL_BUFF_ERROR)
  {
    Serial.println("ERROR - SERIAL_BUFF_ERROR");
  }
  else if(report == END_BYTE_ERROR)
  {
    Serial.println("ERROR - END_BYTE_ERROR");
  }
  else if(report == CHECKSUM_ERROR)
  {
    Serial.println("ERROR - CHECKSUM_ERROR");
  }
  else if(report == TIMEOUT_ERROR)
  {
    Serial.println("ERROR - TIMEOUT_ERROR");
  }
  else if(report == PAYLOAD_ERROR)
  {
    Serial.println("ERROR - PAYLOAD_ERROR");
  }
  else
  {
    Serial.print("ERROR - UNKNOWN ERROR OCCURRED: ");
    Serial.println(report);
  }

  delay(500);
}
7

This post will describe the main functionality.

The entire library is a basic wrapper for a class. This single class has both transmitting and receiving logic inside of it through the use of different member functions.

The TX Side:
The data structure that holds all data that can be sent to the receiver is an array of 16-bit integers. This array is called "outgoingArray" and has a fixed length of 20. In order to send any of the values in this array, a few things must happen:

1.) Identify which index in the array corresponds to the value to be sent. Insert the new value into the array at that index. Repeat for all values the user desires to send in the next packet.

For instance, on my plane, I might want to constantly send 3 data values: elevator servo angle, aileron servo angle, and rudder servo angle. In order to keep track of which value corresponds to which servo angle, I make a convention. I might say elevator servo angle is always stored in outgoingArray[0], aileron servo angle is always stored in outgoingArray[1], and rudder servo angle is always stored in outgoingArray[2].

Note that "outgoingArray" is a public array and is accessed directly.

2.) Create a boolean array with 20 elements (the size of "outgoingArray" - defined by DATA_LEN). At each index, insert a "True" if you want to send the 16-bit value in the "outgoingArray" at the same index - false where the 16-bit value should not be sent.

For instance, if I only want to send the aileron servo angle for the next transition, I would make the following array:

bool outputMask[] = {False};
outputMask[2] = {True};       //aileron data is at index 2 in outgoingArray, all other indices are false and so that no other data will be sent except the aileron data

3.) Pass that array to "sendData()" and watch the desired values show up on the receiving Arduino.

The RX Side:
1.) Periodically call "getData()" to poll the serial buffer and see if data is present for parsing. If data is present, one of 5 reports will be returned, 0 if no data was present. Here is the list of error returns:

//incoming serial data/parsing errors
#define NO_DATA			0
#define SERIAL_BUFF_ERROR	-1
#define END_BYTE_ERROR		-2
#define CHECKSUM_ERROR	        -3
#define TIMEOUT_ERROR		-4
#define PAYLOAD_ERROR		-5

If data was successfully parsed, the function "getData()" will return a 1.

2.) If data was successfully parsed, data in "incomingArray" has been updated. It is up to the user to figure out which indices were changed. It is also up to the user to define which index corresponds to which type of data value - BUT it must remain consistent with which index corresponds to which type of data value in "outgoingArray".

For instance, if the aileron servo angle to be transmitted is always stored in outgoingArray[1], then the aileron servo angle that was received must always stored in incomingArray[1].

Note that "incomingArray" is also a public array and is accessed directly.

8

Power_Broker:
Hmmm, I do have a basic readme

This is the only text I see in the ReadMe

AirComms
Robust Arduino serial data transfer library optimized for speed and accuracy

Quickly and reliably transfer up to 20 16bit values from one Arduino to another
works with hardwired serial connections or with wireless radios such as XBees or bluetooth modules
Library works with with hardware and/or software serial ports
Check out the examples file for details on how to use this library

By documentation I mean something equivalent to this from the AccelStepper library - and it has some serious gaps!

In my experience examples are no substitute for a broad explanation of all the library functions.

...R

9

A nice library, often I have to write code with serial communication, this makes it easy and fast to implement.

Some points that I find limiting factors:

  • Maximum data length of 20
  • Data must be uint16_t and assigned manually
    What if I want to send a struct containing multiple fields? I must worry about the size and assign manually to outgoingArray / from incomingArray using memcpy, for example.

Do you consider making it more generic, like EEPROM.read() and EEPROM.get(), where you can pass any data type and let the function handle it for you?

10

giova014:
A nice library, often I have to write code with serial communication, this makes it easy and fast to implement.

Thank you!

giova014:
Some points that I find limiting factors:

  • Maximum data length of 20
  • Data must be uint16_t and assigned manually

I absolutely agree. However, the data length can be edited by the user to hold more or fewer elements, but I know that can be difficult for new users and a pain for everyone else. I feel that most users for the vast majority of projects don't need more than 20 data fields.

I also agree with the manual assignment of the data array. While this may be less user friendly, it also provides a lot more control over handling the data.

Again, definitely valid points, but most of the functionality and design of the library was based off my project needs. I needed a quick and flexible way to send 16-bit values with a 100% guarantee that the data received was in fact the data originally sent. Any errors in communication will likely result in my plane falling out of the sky.

If enough people are interested, though, I would be willing to write another library - one that is simpler, easier to use, yet still robust.

All: Please reply below if you are interested in this.

11

I didn't explicitly say this in the intro post, but another cool feature of this library is that there is a timeout in-case only part of a packet arrives and the rest doesn't.

12

Hi Robin2,

I just got a link to your post from sterretje, and read it. Thanks for making this library, it appears to be just what I need to both access my DDS Board and my Laptop through Arduino assuming it runs on Arduino. I'm intending to build a software controller that runs on my laptop that sends and receives data on the fly to and from my DDS Board which is controlled by the Arduino code. Since I haven't done much coding on this yet I have only 2 questions about the what language the Library is written in and where does it run?

I'm Looking forward to starting this project finally.

13

pamam:
Hi Robin2,

...Thanks for making this library

I'm glad you like the library, but I think you misunderstood who the lib author is :slight_smile:

Good luck with your project and feel free to ask questions here if you need anything!

14

Power_Broker,

Sorry about that, this forum has a lot of input and I must have gotten confused. So thank you for writing it.

I have managed to get the different libs on serial data into 3 separate word docs so I can study them.
Thanks,

Mel Pama