Yun bridgeclient - receive mailbox messages from python

On my Yun I want to have my Arduino sketch and my Linux python script have bi-directional communication using a message queue. The mailbox support of the bridgeclient would seem to be exactly what I need, with the exception that the bridgeclient apparently doesn't support mailbox reads.

The mailbox class defined here: http://www.arduino.cc/en/Reference/YunMailboxConstructor is a typical example of a message queue interface. Seems to work great for the Arduino, but obviously doesn't apply to the Linux/python side of things.

The bridgeclient.py library has good support for the basics of the Bridge library, but seems to fall short in implementing a complete message queue interface for the mailbox. The only mailbox capability that seems to be supported is a method to write messages to the queue.

What am I missing here? Why isn't there a complete interface for mailbox within the bridgeclient.py library? Specifically, I'd like to be able to check of there are any messages available and then read them.

lakesidebj:
::::SNIP::::

Why isn't there a complete interface for mailbox within the bridgeclient.py library? Specifically, I'd like to be able to check of there are any messages available and then read them.

@lakesidebj,
if one takes a minute to read the Overview for the Arduino Yun, one will notice two processors -- the ATmega32u4 (datasheet - pdf) and the Atheros AR9331 - wikidevi. The latter, the Atheros AR9331 is a full CPU, BUS and GPIO lines. The former, the ATmega32u4, is an 8-bit AVR RISC-based microcontroller featuring 32KB self-programming flash program memory, 2.5KB SRAM, 1KB EEPROM, and more.

It should be apparent - for the ATMega32u4, there is not enough capacity to run a queue. And, if such a queue was constructed, there would be little room left to run anything on the ATMega32u4.

So perhaps the moniker of mailbox may seem a bit more apropos that mail system. Personally, I think mailslot would have been a better name, but I did not write the API.

Respectfully
Jesse

lakesidebj:
On my Yun I want to have my Arduino sketch and my Linux python script have bi-directional communication using a message queue.

I don't know if this works for you, but what I do in my Yun is to share data between Atheros and Atmega32U4 using bridge.put('something','value') and bridge.get('something'). One side must poll the variable and the other side just stores it, this can be done on both directions.

jessemonroy650:
It should be apparent - for the ATMega32u4, there is not enough capacity to run a queue. And, if such a queue was constructed, there would be little room left to run anything on the ATMega32u4.

Just like the key/value database used by the Bridge.put() and Bridge.get() calls, the actual storage for the mailbox message queue is stored on the AR3391 processor and managed by the Linux code. The '32U4 need only keep the current message in memory, and request the next one from the Linux side as needed. This is essentially the same organization as Bridge.get() where the value is retrieved from the Linux side as needed.

While it sounds promising, I haven't had much luck actually using the mailbox mechanism. I can run the example sketch where an http call received by the Yun is used to write to the message queue, and the sketch reads from the message queue, but I haven't been able to do anything else with it (not that I tried real hard.) I was not able to figure out how to make Python code on the AR3391 write to the message queue, nor was I able to figure out how to access data that the sketch wrote to the message queue, either from Python or a remote URL. It seems like a potentially useful mechanism, but I have not yet been able to make use of its potential.

mart256:
I don't know if this works for you, but what I do in my Yun is to share data between Atheros and Atmega32U4 using bridge.put('something','value') and bridge.get('something'). One side must poll the variable and the other side just stores it, this can be done on both directions.

The problem with this is that the receiver has no idea when the sender has set a new value, other than seeing a change. If the sender puts the same value as is already stored under the key, the receiver will have no idea that this happened. Furthermore, if the sender sets a value before the receiver has read the previous value, the previous value will be lost.

The Bridge put/get mechanism works well for some situations, such as a thermostat: you can put() a set point, and get() a current temperature. It doesn't matter how often either value has been set, only the most recent value is important.

But in other applications, it may be important that duplicate values be detectable, and that fast bursts of data not get lost. For example, I have an application with a bar code scanner, which is where I wanted to use the mailbox message queue. The Linux side would read the USB scanner, and send the data to the sketch. Reading the same barcode twice must be possible. When I couldn't get the queue to work, I used put() to send the value. I got around the duplicate value problem by having the sketch put() a blank value after reading the scanned value. This allows it to see the next scan, even if it is the same. But it's an ugly kludge, in that it doesn't solve the burst of input data issue, in fact it actually makes it worse by increasing the time window where a value can be lost.

lakesidebj:
...
The bridgeclient.py library has good support for the basics of the Bridge library, but seems to fall short in implementing a complete message queue interface for the mailbox. The only mailbox capability that seems to be supported is a method to write messages to the queue.

What am I missing here? Why isn't there a complete interface for mailbox within the bridgeclient.py library? Specifically, I'd like to be able to check of there are any messages available and then read them.

ATmega32U4 Mailbox.writeMessage, AR9331 Read (receive mailbox messages from python).

http://forum.arduino.cc/index.php?topic=320763.msg2224137#msg2224137

ShapeShifter:
Just like the key/value database used by the Bridge.put() and Bridge.get() calls, the actual storage for the mailbox message queue is stored on the AR3391 processor and managed by the Linux code. The '32U4 need only keep the current message in memory, and request the next one from the Linux side as needed. This is essentially the same organization as Bridge.get() where the value is retrieved from the Linux side as needed.

::::SNIP::::

@ShapeShifter,
apparently, I am presuming too much. Apologizes to the original poster.

For simplicity, I will use byte in the following description as a common unit; but I could just as easily say char or int. In addition, I will make the same premise for inspecting a queue. That premise is:

A system in which one could query for a possible item requires a hold mechanism.

That is, if you are asking, "if there is something there", then I must put that something somewhere - until you ask if I have it. (This differs from asking for it, but I will cover that.)

When creating a mailing system, as was alluded to, it would be ideal to have the proper conditions. The system, in this case, is in-fact a double queue. The interfaces are equal and balanced, not just programtically, but in terms of resources. That means the ATMega32u4 and the AR9331 are equal; and they must be for this system to work cleanly.

Also, we must conclude there is no out-of-band means to do this; such as an extra electrical wire or circuit.

== the first implementation ==
A system in which one could query for a possible item requires a hold mechanism. And as such, it requires a queue because if one byte is waiting, another is commonly just behind. This can happen for a variety reason, but generally this type of system accomodates an over-run flag, which makes the system useful, but a bit limiting.

To be clear, we have one byte slot for waiting, one byte slot for queueing and a one byte flag. Next we will consider the same system, but with two bytes.

For the moment, I'll ignore the code to make this run.

== the second implementation ==
A system in which one could query for a possible item requires a hold mechanism. This time instead a holding queue, we use a flag - generally the simplest concept being a flip-flop. This flag would work one of two ways. #1 It could trigger an interrupt and the CPU could immediately (or very soon) get the byte. #2 It could just be a simple flag that could be polled. In which case, the loop that checks the flag, gets the byte.

In either case, this fails because the interface to the bridge is non-blocking and time-dependent, and there is no physical control line (no out-of-band line). In other words, this case has no flow control. So, for something like this (the mailing system) to work, it requires flow-control.

== third implementation ==
A system in which one could query for a possible item requires a hold mechanism. Since we are using a bidirectional queued system, flow-control must be accomplished somehow.

At this point, we consider XON/XOFF control (in-band software flow control). For systems using the ASCII character code, XOFF is generally represented using a character or byte with decimal value 19; XON with value 17.

This has one major problem. If we take two bytes out of the ASCII character set and makes them the flow control flags, then we essentially have a hole in our data. We could change the characters to something else, but it would have to be acceptable to our implementation. We'll say that this is acceptable.

== Back to the problem ==

So, let's look at the psudeo-code for this.

if (byte_available_for_bridge) {
 get_byte
 if (output_port_empty) {
 put_byte_into_output
 } elseif (hold_port_empty) {
 put_byte_into_hold
 } else {
 raise_overflow_flag
 }
}

Next, if we have an overflow flag, that is a third state we need to transmit to the other side. The three (3) states being:

  1. have data
  2. have no data
  3. data is over-run

We could get rid of the overflow_flag and keep a tight loop on both sides. And since we created a simple two byte value hole in our data set to handle flow control, this could work. But we'll still need to signal the other CPU that data is available.

In other words, in software we have flow control, but no method of signalling that data is available.

At this point the choice is, get the data quickly or have a hole in your data set.

The alternative, what was implemented, is:

if (byte_available_for_bridge) {
 get_byte
 put_byte_into_output
}

The one case I intentionally did not cover, is a double byte sequence. In this case, the second byte carries the state information of the other CPU/MPU. The downside is this cuts the bandwidth in half.

I'm not sure I covered the most important cases, or all the cases. I'm not entirely sure I was clear either, but I hope this helps.

On this part, I will be very clear. The serial port used with the bridge is a two wire serial port, and nothing more. And we could have a signal for flow control, or data over-run, or both, but at the sacrifice of a hole in our data set.

Jesse

Jesse, I think you're overthinking it. Your comments are quite applicable if you're faced with the prospect of creating a communications protocol to send messages in both directions over a raw serial port. But the authors of the Bridge library have already developed that communications protocol which supports several simultaneous data channels, and out of band signaling, over that simple two-wire serial port. They have also created classes on each side to simplify access to those data channels:

  • Bridge.get()/Bridge.put() - a simple key/value data store
  • YunServer: a listener for incoming TCP connections
  • YunClient: an incoming our outgoing TCP stream
  • HttpClient: an outgoing HTTP request (wrapper for curl)
  • Process: Linux function calls and communications
  • etc.

While you may not agree with the Mailbox name, and/or may be reading more into its functionality, it is a similar service to the above in that it is designed to operate on top of the existing Bridge protocol. The guts of the service are already in place, with the majority of the data handling on the Linux side. It is not necessary to reinvent the wheel here and start from scratch.

The sketch side code is already in place to read from the inbound queue and write to the outbound queue. The HTTP server side code is in place to receive messages via an HTTP request and place them in the queue. What is missing is any sort of Linux side interface code to allow reading or sending messages from Python. I think that cripples the potential usefulness of the service. In his link above, sonnyyu references some code to allow reading these messages by directly accessing the appropriate port to get to the mailbox queue. That's the same sort of mechanism that is used for the other Bridge services, but those services do not require direct port access and parsing of the results: the interface code is already provided.

It's unfortunate that the Python code to do the same with the Mailbox queues was not finished/provided. I've seen several complaints about the get()/put() functionality (things like lost updates or being unable to send the same value twice) that would be solved if the Mailbox implementation were complete.

ShapeShifter:
Jesse, I think you're overthinking it. Your comments are quite applicable if you're faced with the prospect of
::::SNIP::::

@ShapeShifter,
weeellll. not over thinking, but too many steps ahead.

The point I should have made clearly at the beginning is that it is very difficult to create a complete queueing system given the resources. AAMOF, and as you know, serial ports are generally applied with additional buffer. On that point, if I recall right the NS 16550 UART was implemented with the exact type of buffer scheme I used as an example. That is, a two byte buffer, with a set of flags (likely flip-flops).

The presumption I see made will not work well. That is, getting the Linux side to handle the buffers and state-machines for both sides of the serial port.

I should let the OP ask questions again, and will refrain from further comment.

WTF!?! The system is rating God Level. :astonished:

Jesse

Thanks for the input. Driven by the various comments I've reviewed these various bridge and mailbox classes. I'd initially assumed that the mailbox class available to the Arduino side was already a message queue implementation, but it's really just some helper classes surrounding a socket library.

The Arduino side has 'Mailbox' classes that allow us to peek at what is on the socket, messageAvailable(), classes to write to the socket, writeMessage(), and classes to read from the socket, readMessage().

So I get now that the 'Mailbox' class is not a message queue. I was, and still am, confused why the bridgeclient.py library only supports a 'Mailbox' class to write to this socket. I'm not sure of a use case where the Arduino side would need read/write capability to this 'Mailbox' socket, but the Linux/Python side would only need to be able to write the 'Mailbox' socket. So I guess I'm still confused why the two sides don't have a common set of classes.

lakesidebj:
So I guess I'm still confused why the two sides don't have a common set of classes.

Same here. It appears to be a potentially useful construct, but one that is not fully realized.

Yup,

An ability to read the mailbox from python would indeed be handy. Thankfully for what I'm working on I don't need that at the moment but one does have to wonder why one would want to write to the mailbox on the atmel side of life,... if the atmel is the only place where said message could be read again. @_@

Still, since we are talking about this. For future folks visiting, so this is all in one place:

Writing to the mailbox:

Web:-

http://<YUN ADDRESS>/mailbox/my_amazing_message


Python:-

import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')
from bridgeclient import BridgeClient
client = BridgeClient()
client.mailbox("my_amazing_message")


Atmel:-

#include <Mailbox.h>

... next lines in your functions...

Bridge.begin();
Mailbox.begin();
Mailbox.writeMessage("my_amazing_message";

Reading from the mailbox (only available on atmel):

Atmel:-

#include <Mailbox.h>

... next lines in your function...

Bridge.begin();
Mailbox.begin();
String message;
Mailbox.readMessage(message);