CmdMessenger 3 - serial library for Arduino and .NET/Mono, with charting example

Hi,

For a small project I'm working on, I wanted to communicate with a C# application to log some data. Since the serial-over-USB connection is the de-facto connection to the Arduino, I had expected to find a messaging protocol over the serial port without any issue. This turned out to be more difficult than I had expected. On the Interfacing with software page on the Arduino Playground, I found two mature, open and well maintained libraries, Bitlash and Firmata, which seemed to be promising at first. However, these are not quite what I am looking for: both libraries are designed to place all Arduino control at the PC side and make the Arduino into a IO component, and this is not what I want.

When looking for a library that does messaging only, I found the best option to be CmdMessenger 2. It implements

• Commands that can be sent or received
• multiple arguments can be sent together with the command
• Callback functions can be triggered on received commands

But even though it was already a pretty good piece of work, it was still lacking in some aspects. I modified so that:

• It has an more symmetrical implementation: It is now possible to send all basic data-types (char arrays, floats, ints, bytes) that could already be received in v2, instead of just strings.
• It can wait for an acknowledge command after sending
• It has the ability to escape data. Special characters that would otherwise be interpreted as field separators or command separators are can now be escaped and harmless.
• It has the ability to send binary data. Using escaping, basic data-types can now be send over in binary form, for efficiency and accuracy

But most importantly, I wrote an additional implementation of the library for C# that runs both on .NET and Mono. I gathered that for a messaging protocol to work well, both sides would need to speak it fluently

The package can be downloaded in different ways:

• Using the Arduino Library Manager
• Using the PlatformIO package manager
• By cloning the GitHub repository
• By downloading a zipfile with all dependencies

A more detailed explanation on installing can be found here:

My ridiculously expansive blog posts on how to use the library can be found here
http://thijs.elenbaas.net/tag/cmdmessenger/

I tested the code with MonoDevelop on Windows 7, but I'm curious to hear if it also work well on Linux and OS X, so let me know if you have it running or are experiencing issues.

I constructed a simple example: the PC will toggle the on-board led of the Arduino, by sending commands to set the state of the led. The Arduino receives the command, reads the parameter with the requested led state and set the led accordingly. Next, the Arduino will send a command to the PC returning the set state.

First we need to define the commands. At Arduino side:

enum
{
  kSetLed , // Command to request led to be set in specific state
  kStatus , // Command to report status
};

And at the PC side:

enum Command
{
  SetLed, // Command to request led to be set in specific state
  Status, // Command to report status
};

Note that these should be the same commands, in the same order. Of course you can follow language dependent coding conventions, resulting in (slightly) different naming. On the Arduino side we need to the listen to the SedLed command. We do this by attaching a callback function to this command ID:

void attachCommandCallbacks()
{
  // Attach callback methods
  cmdMessenger.attach(kSetLed, OnSetLed);

void OnSetLed()
{
  // Read led state argument, interpret string as boolean
  ledState = cmdMessenger.readBoolArg();
  // Set led
  digitalWrite(kBlinkLed, ledState?HIGH:LOW);
  // Send back status that describes the led state
  cmdMessenger.sendCmd(kStatus,(int)ledState);
}

On the PC side we will listen for a response using the Status command :

private void AttachCommandCallBacks()
{
   _cmdMessenger.Attach((int)Command.Status, OnStatus);
}

// Callback function that prints the Arduino status to the console
void OnStatus(ReceivedCommand arguments)
{
   Console.Write("Arduino status: ");
   Console.WriteLine(arguments.ReadStringArg());
}

That's all there's to it for basic 2-way communication. The library comes with generated documentation for both platforms, and the API for the Arduino is explained on the playground, so enough to get you started.

I'll try to add a more extensive example later.

Hi, I've gotten some questions on how to make a windows application with a graphical user interface, so I wrote 2 examples that show how to use CmdMessenger with GUI applications. I had to make changes to the library to make it play nicely with WinForms, such as invoking messages on the UI thread.

The first example receives data from the Arduino AnalogPins and plots this data in real-time to a moving-window chart. You could easily extend it to a more capable datalogger.

561×504 25.2 KB

The second example allows you to change the blink frequency of the on-board led using a slider.

Both examples work under .NET as well as Mono

A few months ago I started this thread to show the changes I made to the CmdMessenger library. I got a lot of positive reactions on my blog as well as a lot of e-mails. Thanks for your support!

I also got a lot of questions about existing and non-existing functionality. I implemented a lot of new stuff, around the themes: support for more devices, more forms of communication and support for a wider range of speeds.

You can still download the library here:
http://thijs.elenbaas.net/downloads/?did=9

Or at the Github repository:
https://github.com/thijse/Arduino-Libraries/tree/master/CmdMessenger

And find a more detailed explanation of the changes here
http://thijs.elenbaas.net/2013/11/arduino-to-pc-messaging-more-devices-protocols-speeds/
and
http://thijs.elenbaas.net/2013/09/arduino-to-pc-messaging/

Transport Layers

While the Arduino CmdMessenger library was already prepared for different transport layers (just inject it as a stream), in the C# version the transport layer was not nicely separated from the communication protocol. I fixed this, and now you can easily create a new mode of transport and inject it in the library, if it implements the following interface:

public interface ITransport
{
	int BytesInBuffer();
	byte[] Read();
	bool StartListening();
	bool StopListening();
	void Write(byte[] buffer);
	event EventHandler NewDataReceived; 
}

Communication speed

I was contracted by a company that wants to use the library in a quadcopter. I'm no expert on arial communication, but I imagine that depending on things like distance and line-of-sight, the speed may vary and even may become intermittent. At the other side of the speed spectrum is the Teensy 3. Like the Arduino, the Teensy 3 implements a virtual serial port over USB. However, unlike the Arduino, the Teensy 3 lets the serial port run on full USB speed (12Mbit/s). For this reason I added adaptive throttling (similar to what is described in this blog). This means that at low communication speeds the buffer polling happens at a relatively large interval, reducing system load. If data comes in faster, the polling speeds ramps up (at the expense of a higher system load).

I have tested this with the Teensy 3 of a friend, and it seems to run quite smoothly with CmdMessenger. Sorry, no benchmarks yet.

Command Queues

CmdMessenger has also become more responsive due another significant change I introduced: both the send- and receive-commands are now being queued.
In my previous post I introduced two small applications with a simple GUI (DataLogging and ArduinoController). Both worked with the previous version of command messenger, but I was not really satisfied with either of them: Using ArduinoController I noticed that when I pulled the slider that changed the blinking frequency of a led, the slider moved very choppy, halting and then make a big jump. The reason is that, for every slider movement an event is raised that issues a new command the Arduino. This happened in the GUI thread, and the GUI blocked until this was done.

In the new version the commands that are generated when the slider is moved are now queued, and the UI thread can continue immediately after putting them on the queue . The commands on the queue are sent in the background as fast as the Arduino can receive them. I also implemented a queue for receiving commands, for similar reasons as explained in my blog post.

Queue strategies

Generally speaking, the queues are there to buffer incidental speed differences between the sending- and receiving party. However, if the speed difference is structural the queues will continue to expand and eventually grow too large, even for a PC. The only way to remedy this is by throwing away commands. This leads me to the next improvement:

The main question is: how to decide which commands to throw away? The short answer is: I don't know.

The slightly longer answer is: I don't know, since it depends on the application and the commands in question. Only the writer of the application knows how to deal with this. The good thing is you can decide for yourself by using specific queue strategies, or by even writing your own.

Let me give an example: recall the slider that changes the blink frequency. when we move the slider, the "SliderValueChanged" event is called many, many, times, resulting in a queue full of "SetBlinkFrequency" commands. However, we are only interested in sending the most up-to-date (the last) blink frequency.

For this reason we can wrap the commands in a so-called "CollapseCommandStrategy". This strategy walks through the queue, and if it finds an other "SetBlinkFrequency" command, it will replace it at that position in the queue. If there is no "SetBlinkFrequency" command in the queue, it will add the command to the back of the queue.

The usage is simple:

var command = new SendCommand((int)Command.SetLedFrequency,ledFrequency);
_cmdMessenger.QueueCommand(new CollapseCommandStrategy(command));

For more examples of strategies that throw away stale commands, put commands at the front of the queue instead of at the back, have a look at my blogpost http://thijs.elenbaas.net/2013/11/arduino-to-pc-messaging-more-devices-protocols-speeds/

CmdMessenger Arduino updates

On the Arduino side I two small updates. My thanks go to Nate who suggested these changes

1. Added the ability to send longs:

long readLongArg()

2. Added the ability to see if the last argument fetched is well formed:

isArgOk()

TemperatureController example

I also added an actually useful sample of a PC controlled, Arduino based thermostat. It uses a K-type thermocouple and a MAX31855 / Max6675 breakout board to measure temperature, a Solid State Relay to turn a heater on and off, and a PID controller implementation to steer the temperature optimally (no overshoots) to the goal temperature. My personal aim is to update update or Gaggia Classic espresso machine with temperature stabilization, but this sample set up to be so generic that you can use it for everything: brewing beer, controlling a clay oven, etc.

I recommend starting simple. I bought a simple 10 euro electric water boiler, spliced the power cable and inserted the solid state relay and let the thermocouple just hang in the water.

With the trackbar the goal temperature is changed. Also here, every slider change queues a command to be sent to the Arduino. The top chart shows a scrolling chart of the measured temperature and the goal temperature. The bottom chart shows the control value of the heater, with a value between 0 and 1.

Since the the heater is controlled using a solid state relay, it can only be turned on or off. In order control the heater power we need to modulate the power. Below is a representation of such a Pulse Width Modulated signal with changing duty cycles

We have set our duty cycle time to 3s, in order not switch too often, and we let the PID controller control duty cycle percentage. The resulting PWM cycle is also shown in the bottom chart.

600×550 63.7 KB

Next steps

The library is now flexible enough to work with different hardware platforms, and I would really like to try this out for more platforms. As said before, the library is already running nicely on a Teensy 3. The next step is getting it to run on a wireless device such as the Flutter , Spark Core or RFDuino.

Yes, yet another update of CmdMessenger, now version 3.6! This time, the main changes are: speed optimization for the fast ARM based boards, better compatibility in data formats between the PC and Arduino, and a unit testing framework.

The library can still be downloaded at the location in the first post, both from zip file or at the Github repository. I have written a blog-post about this particular release that can be found here:

http://thijs.elenbaas.net/2014/05/arduino-to-pc-messaging-improved-speed-compatibility-and-testing/

An overview of all articles on CmdMessenger can be found here:

http://thijs.elenbaas.net/tag/CmdMessenger/

Speed improvements and benchmarks

Recently, I received a Teensy 3.1. Looking at the raw receive benchmarks of the board, I was expecting pretty solid results for CmdMessenger. However, my first measurements wher a bit disappointing: the receiving speed was only twice that of an the Arduino Uno.

It occurred to me that while the Teensy is able to quickly receive and process large blocks of data over serial USB, the PC side of CmdMessenger sends only very small messages, typically around 10 bytes.

This means that we could improve performance by sending longer messages. Because the PC CmdMessenger already had a send queue for asynchronously sending commands, I could combine multiple messages on the queue, resulting in a single, long string of commands, making it much more efficient to send. As it turns out, this makes a world of difference, as you can see in the chart below. Compare the buffered and unbuffered send benchmark:

It turns out that it is 30 times faster than the Arduino Uno or Nano. I think that is a pretty excellent result! While there are many good reasons to use an Arduino Uno or Nano, if you want to have quick communication over USB, an ARM board like the Teensy is a very good choice!

Compatibility

An issue I faced with sending commands between PC and Arduino is that the definitions of some basic variables differ. This grew further with the advent of the ARM based Arduino boards. Consider the following variables and their size on the different platforms.

It is very important to use the same data formats on both side of the line. If we use the different formats, this can result truncation and loss of accuracy for clear text values and completely wrong values for binary values.

This has been solved by doing two things:
1.In the library as well as the examples I switched from Int, Long & Short to the unambiguous Int16 and Int32 descriptions.
2. For floating point values the library uses the Float and Double description, but I removed all Single references. Float was already unambiguously 32 bits on all platforms, but in order to process Doubles correctly on all platforms, I modified the library: you can now set on the PC side whether the receiving embedded board is 32 bits or 16 bits:

_cmdMessenger.BoardType = BoardType.Bit32

In the former case, unaltered 64 bit values are sent and received, but In the latter case the CmdMessenger library will automatically convert values. That is, double variables are automatically converted from 32 bit to 16 when being sent to the board, and converted from 16 bit to 32 bit when being send to the PC.

Unit tests

CmdMessenger has steadily grown in complexity on both on Arduino and PC side. As a result it requires a lot of testing with every release. Because of this I started to add unit tests, to perform the test I would normally do by hand. At this point they do not test every scenario I can think of, but they cover all mayor functionality. It runs the following sets of tests:

Test all plain text formats
Test all binary formats
Test sending multiple arguments
Test send and receive speed

As a matter of fact, making and running these tests revealed multiple bugs and shortcomings, ranging from small inconveniences to hidden and serious bugs. These have been fixed in this release, and I believe this CmdMessenger 3.6 is the most robust release so far!

Scientific Format

Using the testing framework, I learned that the Arduino is only capable of printing clear-text floats(and doubles) in the range of from -2,147,483,648 to 2,147,483,647. I have added a print function that will send clear-text values in scientific format that span the full range -3.4028235E+38 to -3.4028235E+38.

cmdMessenger.printSci (arg, n)

So again, please try it out and let me know any feedback that you have.

A post to announce a major release of CmdMessenger, Release 4.0. This is a pretty big one, possible because CmdMessenger development was joined by Valeriy Kucherenko. Many of the new features and improvements in this version CmdMessenger came from him.

Improvements to the code base

Let's start with what is probably the least exciting point from a user perspective: We made some big improvements to the code base. The previous version of the .NET library relied on polling for new data to be available, which resulted in a trade-off between system load and responsiveness. This updates uses inter-thread signalling which is faster, less resource hungry and does not lead to thread starvation.
Furthermore a whole range of improvements where made: improved thread safety, separation of concerns and more. The code has been running 24/7 on a (Mono-Linux!) test system without issues.

Tested Linux compatibility

While I stated in previous releases that the .NET library was MONO compatible and therefore SHOULD run on Linux, this had not been tested. Now Valeriy did! One thing to note is that under Mono on Linux GetPortNames() will never show Arduino ports, so we added our own implementation of this core Mono function.

VB samples

It was always possible to use the CmdMessenger library in Visual Basic. However, being no Visual Basic user myself I had not added examples. But since there so many questions on the forum on VB to Arduino communication, and because I think CmdMessenger can really help, I've translated the C# samples into VB (thanks to the online C#-to-VB translators).

' Commands
Friend Enum CommandIDs
   SetLed ' Command to request led to be set in specific state
End Enum

Public Class Receive
   Private _serialTranspoart As SerialTransport
   Private _cmdMessenger As CmdMessenger
   Private _ledState As Boolean

   ' Setup function
   Public Sub Setup()
       _ledState = False

       
       _serialTransport = New SerialTransport()                 ' Create Serial port
       _serialTransport.CurrentSerialSettings.PortName = "COM6" ' Set com port
       _serialTransport.CurrentSerialSettings.BaudRate = 115200 ' Set baud rate

       _cmdMessenger = New CmdMessenger(_serialTransport, BoardType.Bit16)        
       _cmdMessenger.Connect()                                  ' Start listening
   End Sub

   ' Loop function
   Public Sub [Loop]()
       ' Create command
       Dim command = New SendCommand(CommandIDs.SetLed, _ledState)

       ' Send command
       _cmdMessenger.SendCommand(command)

       Console.Write("Turning led ")
       Console.WriteLine(If(_ledState, "on", "off"))

       ' Wait for 1 second and repeat
       Thread.Sleep(1000)
       _ledState = Not _ledState ' Toggle led state
   End Sub

End Class

AutoConnect & Watchdog

The Library now comes with an optional connection manager: For serial port transport, It will scan all serial ports present, and try them at different speeds. If a serial port responds with the correct answer, the connection is confirmed.

Once the connection is confirmed, a watchdog can be started to monitor the connection. If the connection is silent for too long, the watchdog will ask the connected device for a sign of life. If no response comes, the connection manager will take over and start scanning again. All This makes it very easy to set up and maintain connections. Below is some sample code showing the usage

Setting this up is pretty easy:

// You want to be sure you are connecting with the correct device.
// Identify your device, for example, by using a GUID
private const string CommunicationIdentifier = "BFAF4176-766E-436A-ADF2-96133C02B03C";

...
public void Setup()
{
// Use serial transport
_transport = new SerialTransport();

// This is a 16-bit Arduino board
_cmdMessenger = new CmdMessenger(_transport, BoardType.Bit16);

// The Connection manager can store connection settings, in order to reconnect more quickly next time 
var serialConnectionStorer = new SerialConnectionStorer("SerialConnectionManagerSettings.cfg");

// Configure  the connection manager
_connectionManager = new SerialConnectionManager(
_transport as SerialTransport,      // Transport layer
_cmdMessenger,                      // CmdMessenger instance
(int) Command.Identify,             // Command to ask Arduino for identification
CommunicationIdentifier,            // Expected answer
serialConnectionStorer)             // Storage functionality
{               
WatchdogEnabled = true,             // Enable watchdog functionality.
};

// Message if the connection is set up
_connectionManager.ConnectionFound += (sender, eventArgs) =>
{
Console.WriteLine("Connected!");
};

// Message if the connection is set up
_connectionManager.ConnectionTimeout += (sender, eventArgs) =>
{
Console.WriteLine("Disonnected!");
};

// Activate connection manager
_connectionManager.StartConnectionManager();
}
...

Bluetooth

We have added a transport layer for Bluetooth, and tested expensively with the commonly used HC-05 and HC-06 breakout boards. The serial port transport layer will also work with these boards: if you set them up correctly (bind, enable virtual serial port), you can use the virtual serial port to communicate.

We have gone a step further, though. By using the underlying RFCOM layer (using the 32-Feet library), the Bluetooth layer does not need an virtual serial connection and will immediately run on the correct speed without overhead. Also, it allows for much nicer auto-connect behavior: Your Bluetooth device needs only be in range. The auto-connect code with search for both paired and unpaired devices. If unpaired, it will try the most commonly PIN codes to pair, and subsequently to connect.

The only thing you would have to do is connect your Arduino correctly to the HC-05/06 device, and make sure that the serial speed set in your script is the same as the speed set in your Bluetooth device (by default 9600, but this can be changed through the AT commands).

Bluetooth auto-connect is very similar to the serial connection auto-connect functionality:

_transport = new BluetoothTransport()
{
 // If you know your bluetooth device and you have already paired
 // you can directly connect to you Bluetooth Device by address.
 // Under windows you can find the address at:
 //    Control Panel >> All Control Panel Items >> Devices and Printers
 //    Right-click on device >> properties >> Unique id
 CurrentBluetoothDeviceInfo = BluetoothUtils.DeviceByAdress("20:13:07:26:10:08")
};

_cmdMessenger = new CmdMessenger(_transport);

// Bluetooth connection storer
var bluetoothConnectionStorer = new BluetoothConnectionStorer("BluetoothConnectionManagerSettings.cfg");

// Set up connection manager
_connectionManager = new BluetoothConnectionManager(
 _transport as BluetoothTransport, 
 _cmdMessenger,
 (int) Command.Identify, 
 CommunicationIdentifier,
 bluetoothConnectionStorer)
{
 // Enable watchdog functionality.
 WatchdogEnabled = true,

 // Add PIN codes for specific devices
 DevicePins =
 {
 {"01:02:03:04:05:06","6666"},
 {"01:02:03:04:05:07","7777"},
 },

 // Add PIN code to try on all unpaired devices
 // (the following PINs are tried by default: 0000, 1111, 1234 )
 GeneralPins =
 {
 "8888",
 }
};