Best way to exchange data between arudino and linux for IoT

Hello,

I'am developing an application on an arduino Yun, that receive data from several sensors and has to send them via cURL to a context broker. I would like to have some explanations about the different ways to exchange info between the two cores of the Yun. I am ysing the Bridge class right now but I have to put a lot of data with a lot of different keys. I don't know if it is bad but it sure sounds bad. I was checking the Mailbox class but i can't really get to understand how it actually works. What are the best practices, there is a standard or something?

I've not used the Mailbox class much, haven't found a need for it.

The Bridge.put() and Bridge.get() methods have some serious drawbacks in my mind, at least as far as data communications go. They are slow, and there is no way to know that a value has been set, the only option is to constantly poll the keys and look for changes. That means that if the same value is written twice, there is no way to know it. This method works if the data something where pnlybthe current value is important, like a thermostat setpoint, but it is limiting if you want every data sample to go through.

My go-to method of communications between the two sides of the Yun is the Process class. It works exactly like a direct serial connection between the two. It has the disadvantage that it's not quite as fast as a raw serial connection, but has the advantage that you can have several Processes running, each with their own independent data stream, and you can still use other Bridge Library functions if needed.

I like to put as much of the processing down in the Linux side as possible, and just have the sketch be a fairly dumb I/O processor. The sketch has a global instance of a Process() object that launches the Linux side script asynchronously and serves as the data path. As data is collected, it is written to the Process object, using the same methods you'd use with the Serial class. On the Linux side, the script reads from STDIN and parses the data, does what it needs to do, and writes any response data to STDOUT. The sketch can read this data from the Process object, just like it would from the Serial class.

It's important that the Linux side script be running in unbuffered mode. For example,
Python normally runs in a buffered mode where output data is buffered up until there is a large amount, then it sent as a big block. If the amount of output data is low, it will seem like it's doing nothing for a long time, then suddenly the sketch will get a large amount of data at once.

For Python, you can do this by including the "-u" option a couple different ways:

• Put it in the shebang as the first line of the script, after the Python executable name: "#!/usr/bin/python -u" - Use this method if you simply pass the name of the script to the Process object, for example: "process.runAsynchronously("/mnt/sda1/myScript.py);"
• Explicitly add the option in the runAsynchronously call, for example: "process.runAsynchronously("/usr/bin/python -u /mnt/sda1/myScript.py);"

ShapeShifter:
My go-to method of communications between the two sides of the Yun is the Process class. It works exactly like a direct serial connection between the two. It has the disadvantage that it's not quite as fast as a raw serial connection, but has the advantage that you can have several Processes running, each with their own independent data stream, and you can still use other Bridge Library functions if needed.

I like to put as much of the processing down in the Linux side as possible, and just have the sketch be a fairly dumb I/O processor. The sketch has a global instance of a Process() object that launches the Linux side script asynchronously and serves as the data path. As data is collected, it is written to the Process object, using the same methods you'd use with the Serial class. On the Linux side, the script reads from STDIN and parses the data, does what it needs to do, and writes any response data to STDOUT. The sketch can read this data from the Process object, just like it would from the Serial class.

This sounds interesting, but i can't really see how i can use it. For what i am uderstanding i am not supposed to start a particular process, just use the process object to read/write?

Bshow:
For what i am uderstanding i am not supposed to start a particular process, just use the process object to read/write?

I'm sorry if it wasn't clear.

What I do is create a global instance of a Process object at the top of the sketch.

In the setup() function, I call runAsynchronously() method of that process object to start the script on the Linux side.

In the loop() function, I write samples to the process object, using the same techniques that I would do for a serial port connection. I use a format that minimizes the number of characters sent, and yet makes it easy to parse the data on the Linux side. Generally, this consists of print() and println() calls.

Also, in the loop() function, I look for and process and data from the process object. Generally, this consists of using available() and read().

The general idea is that the process object is initialized once, and stays open and running. This launches the script on the Linux side, and allows continuous communications between the sketch and the script. Communications are handled in exactly the same way that one would do communications over a serial port - this is possible since both the Process class and the Serial class derive from the Stream class, which is the class that gives the print(), println(), available(), read(), and other communications related functions.

For robustness, I usually also include a check in the loop() function that calls the running() method of the process object. If the Linux script should crash or terminate, this function will return false. In that case, I call the runAsynchronously() method to restart the script.

I like this. I'd like to see some very basic examples of how you might set up a thermostat to run on the arduino side and display a temp value to a web page, and allow a button press to send a command to change a parameter back on the arduino.

Currently, I have a fairly in depth thermostat set up exactly like this using Bridge.put and get. It works, but for the last 6 months I've been trying to track down why the whole system locks up after 24 hours. Switching to the Yun shield on the mega from a mega with an ethernet shield didnt fix it.

Right now I have about 17 parameters sent to the keystore at different times. Some are sent several times per second, some every 5 seconds and some only when theyre changed.

For commands, the website writes a single char to the keystore which is read each time thru the loop and over written back to 0 when read.

I can tell this is slow and I would like to implement something much more efficient. Any example code would be awesome. Thanks!!

Another question:
Serial.begin(9600) seems to be the default for bridge. Should I increase that to speed things up? Woudn't I have to modify something on the linux side so theyre the same speed?
Its easy to change on the serial monitor but I have no idea how to change it on the linux side.

vulture2600:
I like this. I'd like to see some very basic examples of how you might set up a thermostat to run on the arduino side and display a temp value to a web page, and allow a button press to send a command to change a parameter back on the arduino.

I don't have any existing samples that would show something like this, anything I have is part of a much bigger project - including a snippet of code may not be helpful, and including the whole thing would just be confusing.

You mention running the thermostat on the Arduino side - I would approach a project like this from the opposite direction: put as much logic as possible on the Linux side. All of the control data, processing, and web interface is in one place, and you don't have to move as much data between the two sides of the Yun. The Arduino side would be just a dumb I/O processor, reading temperature and other sensors and sending the raw data to the Linux side, and receiving output states from the Linux side and controlling output signals.

That idea is similar to a project a posted about a while ago: HowTo: Using RRD - Round Robin Database

While that discussion goes into a tangent that is not necessarily applicable to what you want to do, the general architecture could be applicable to your project. This was a very simple lighting controller: it could control a relay to turn a lighting load on and off, and it had a current transformer to monitor how much power was being consumed by the lights. All of the logic to decide when to turn the light on and off, and to monitor the power consumption and determine if there was a burned out bulb or other problem, was all on the Linux side. The communications between the two sides was very simple: the sketch periodically read the analog input and sent the raw value to the Process object using println(), and the Linux side would respond with a 1 or zero value to indicate if the relay should be on or off. It was trivial for me since I had only one value to send in each direction, so the only delimiter I needed was a line terminator. If you have more than one value, you will need to come up with a format for the data: either a fixed number of characters where you just count characters, or use delimiters like commas, or something along those lines. This is basically the same issue you would have if you were using a straight serial connection.

For the web interface, I used the Python based Bottle framework. There are plenty of other options out there as well. Use whatever you are comfortable with. The sketch used a Process object to launch the Python script that ran the Bottle application and also handled the communications with the sketch. All of the control information was local to the Python script, and only the low-level control information was sent between the sketch and the Python script.

vulture2600:
Another question:
Serial.begin(9600) seems to be the default for bridge. Should I increase that to speed things up? Woudn't I have to modify something on the linux side so theyre the same speed?
Its easy to change on the serial monitor but I have no idea how to change it on the linux side.

The following is based on a real Yun board. Using a Yun Shield with some other Arduino board may be different, I have no direct experience with it.

Serial.begin(9600) initializes the USB serial port connection to your computer, and allows talking to the Serial Monitor or other terminal emulator. Since it is a USB connection that is managed directly by the '32U4 processor running the sketch, the baud rate is meaningless. You can use any value, and it makes no difference.

The communications to the Linux side is done using Serial1, which is a different interface that is managed by the Bridge library. You don't have to worry about the baud rate there, as it already uses a quite fast rate. (I don't remember the speed, but it's somewhere over a hundred thousand bits per second, if I recall correctly.)

Thanks for the reply.

My current setup dumbs down the Arduino side as much as I can to perform as a standalone thermostat. All the heavy lifting is done on the Linux side. I just dont know how to make it more efficient. Thats why I asked to see examples of your Processes. I really have no idea where to start and I dont know python yet.