Send/Recive data from YUN /data, /mailbox or REST (project)

Hello

I want to make a webpage (on my domain) to view status of the device, and to be able to change data on the yun, from the same page.

I want to offload as much code as I can to the webpage, and keep yun as free as possible to do other things, not process incoming / outgoing data all the time.

The sending part is easy, I just make a string with the relevant data I want to send to the page, and curl it to a script for processing and then call the Pusher api to update the page.

Now, the reciving part, I'm not sure how to implement it.

I saw on the yun's guide

"/data" is used to access to the internal key/value storage. The available calls are:

/put/KEY/VALUE : stores a value inside the storage

And it seems to work, putting a key/value pair returns a json response.

My question is, can I access that "internal key/value storage" from the sketch itself?
So if I have a delay, to use that internal key as the delay, and to be able to change it via a /data/ call?

Or, what is the best way to do this, reciving commands from a page with YUN?

Nicely phrased post!

chriissis:
My question is, can I access that "internal key/value storage" from the sketch itself?

Bridge.get() will read these values.

Bridge.put() will set these values, so they could be read with a /data/get web call.

Or, what is the best way to do this, reciving commands from a page with YUN?

The above is a good way of passing data, but not commands. The difference is that the previous method just saves the last set value, exactly like setting a variable, without notifying that a change has been made. So, if the same value is set twice, the sketch won't know that the second put was made. It could put() a different value once a command was read, so that it will recognize the next time it's set, but that opens the window to a race condition which can cause missed commands.

For commands, where it's necessary to see every one, even if they come in quickly or are the same as the previous one, a better method might be the Miailbox. You send a command with a /mailbox/xxx command, and receive it with the Mailbox class. this is a queue of messages where each message will be received in the order they were sent.

Another alternative for handling commands and/or data is a REST API as discussed in the Bridge example. this takes more coding in the sketch, but allows a synchronous access method, and lets you define a very rich API.

chriissis:
My question is, can I access that "internal key/value storage" from the sketch itself?

Yes, you can use the put/get methods provided by the Bridge library. With these methods you can access and modify the internal key/value storage on the linux side

Thanks guys (or gals)!

So /data/ is not that great (for this project) since it stores the data on the processor's side.
If I want to read the data and use it, I would have to call the Bridge.get() for every key that I want to use on the sketch, to get the updated value (if there's one), on every single restart of the loop().

That doesn't look right.

I guess the best options is to make my own "API" to handle the REST requests, and extend on top of that bridge example.

So, mailbox or "rest" ? let's vote!

BUT, if I send multiple IP/command/key/value requests, and the microcontroller is busy doing stuff, It will get them in order, or it will drop the requests and it will never get those dropped ones?

Or maybe I combine the "API" with the mailbox thingie, and send the messages in form or a REST request string, and parse that string to see what's there.

chriissis:
So /data/ is not that great (for this project) since it stores the data on the processor's side.
If I want to read the data and use it, I would have to call the Bridge.get() for every key that I want to use on the sketch, to get the updated value (if there's one), on every single restart of the loop().

To be clear, the data is stored in the RAM of the linux processor on the Yun. Yes, to read the value, the sketch will have to go over the bridge and fetch the value from the Linux side.

It's not as bad as it seems, you still only have to code the get() once (for each value) at the top of loop(), and loop() will take care of reading them over and over. The only real concern is if you have critical timing in the loop that requires very fast repetition.

That doesn't look right.

I don't know, polling is a very valid paradigm. It sounds repetitive, but microprocessors are used to that, it's their fate to be doing the same thing over and over.

I guess the best options is to make my own "API" to handle the REST requests, and extend on top of that bridge example.

That would be valid. The data would be stored in the sketch, so the subsequent polling of the value would be done in local RAM. The downside to this is that the request becomes synchronous: the caller sending the request must wait while the sketch gets around to accepting the connection, receiving and processing the request, and sending any response, and closing the connection. With the /data/ and /mailbox/ methods, it's all handled by the Linux side and the sketch is not involved until the sketch polls the Linux side for the latest data - it's much more asynchronous. (Synchronous vs asynchronous each have their pros and cons, and a lot depends on the application as to which is better.)

BUT, if I send multiple IP/command/key/value requests, and the microcontroller is busy doing stuff, It will get them in order, or it will drop the requests and it will never get those dropped ones?

Using mailbox, it should get them in order, unless they are coming in so fast they are swamping the Linux processor at which point some may be dropped. With a REST API, the sketch will get them in order, unless they are coming in so fast the sketch is swamped, at which point some may be dropped. The difference is that the Linux side is much faster than the sketch side, so I guess it's less likely to get overruns with the Mailbox or data/put mechanisms?

I've not done relative timing measurements, so I can't tell you which method gives the best performance, or under which conditions they may break down. But I have implemented a sketch that uses Bridge.get() to repeatedly poll the Linux side for several data items, and it has worked reliably for weeks at a time (the time limit being when I unplugged the Yun, not because of any failure.) Unless you have very tight loop() execution requirements, don't be too quick to dismiss Bridge.get(). And if your timing requirements are that tight, polling for Mailbox data, and even processing REST API requests may be too much time as well.

Thanks guys for all your great answers.

I think the /mailbox is the best options for what I need.

I will send the commands in form of /mailbox/ID/and/the/string/to/be/parsed

And after parsing the string, It will send a confirmation with curl [...]/ID/and/the/response/string

That way I can have a log on the page with the command and the response paired by the ID, and see if something is dropped and the response of the command (I can send the status of the command, the value it set, or anything I wish to send back) and if something's wrong I can check the log and see where it went wrong

Now, the fun part starts, writing the code.

That should work.

I just want to point out that if you use the REST API method, the response is almost automatic: just write to the YunClient before closing it, and the sender of the original command will get that in the response. One bi-directional operation instead of two separate operations, no worries about what to do if the response curl request fails, and no need to remember the IP address of the server to receive responses. The advantage is that it gets all of the communications gathered in one cohesive spot using a single synchronous connection. The disadvantage is that it loses the asynchronous buffering which may be an issue if the data is bursty with periods of quickly occurring commands.

Edit

I managed to install a spacebrew server of my own and create on the arduino side one publisher and one subscriber, and on the page too, one publisher and one subscriber.

And I can "link" the page - yun via publisher - subscriber to send message from one to another

This way I don't need to know the IP of Yun, and the string transfer seems almost instant (with the rest / mailbox there is ~300-600 ms delay

Sounds very interesting, especially the part about installing your own server. Can you share some more details?

I did a slight amount of reading about spacebrew, but quickly dismissed it because I don't want to keep going out to a server on the Internet for everything, I need a stand-alone solution that works on the local network. Perhaps a local spacebrew server would meet my needs.

And I'm sure it would be interesting to other people as well.

Well, it's not really my own, I use the heroku service to run the spacebrew "server". So it's over internet, not local.

But, if you have a "server" that can run nodejs apps, you can run your spacebrew server on that.

  1. Download and install Node.js
    Clone the repo from github
    Install the dependencies using node packaged modules
    websockets module: npm install ws
    forever module: npm install forever-monitor
  2. Run the Server
    Open terminal and navigate to the base directory of the spacebrew server
    Run the server by using node node_server_forever.js

And that's all, you can point your arduino(s) / page to that server when connecting, and it should work.

And the links between page and arduino.

Hmmm... the Yun can run Node.js. I wonder if it has enough horsepower to run the server? One "master" Yun could serve as the switchboard for others on the network.

On the other hand, I've got a Beagle Bone Black sitting next to me that I haven't had time yet to open and fire up. That has more horsepower and could probably handle it better. I'll need to experiment with that. (But if I haven't even had time to open the box, where am I going to find the time to play with it? ::slight_smile: )