Beginner's Introduction to Yun Web Services

Official Hardware Classic Yún
1

This page is mostly a response to a recently posted long multi-part question which was asked in an old thread. Since the question is not really related to that other thread, and since the answer is likely to be interesting to a wider audience, I'm starting a new thread on the topic.

Note: this reply has many instances of URLs that access the Yun. I will use the address "arduino.local" since that is the default network name for a new Yun. The "arduino" portion of the name can be changed during the initial setup of your new Yun, in which case you should substitute your Yun's name. For example, if you named your Yun "MyYun" then you would want to use "MyYun.local" wherever this discussion uses "arduino.local". Also, some computers don't have the necessary software to be able to decode these names, so you may have to substitute the IP address of your Yun, using something like "192.168.1.123" in place of "arduino.local".

jumpjack:
I can't find in the sample page a descritpion on how the web page must be set up to send proper commands to the sketch, any help?

You're probably having trouble because there are many ways to do it. Which method is best depends you your individual goals. More discussion on that later, after I address a few more questions.

Additionally, I don't understand when the page talks about "REST" (I know POST and GET) and the required password...

WikiPedia: REST

That's a reasonable definition of REST, but probably doesn't help you much. The easiest way to think of it is that you are performing an HTTP GET operation with a URL that can be easily parsed. When the Yun receives any request that has a URL that starts with [color=blue]http://arduino.local/arduino/[/color], the rest of the URL string is passed to the sketch via a YunServer/YunClient object. For example, when running the Bridge Example, an incoming request of [color=blue]http://arduino.local/arduino/digital/13/1[/color] will pass the string [color=blue]digital/13/1[/color] to the YunClient object, which will be decoded by the sketch and end up calling the function digitalWrite(13,1).

The Bridge Example implements a fairly simple and generic API. You are free to define any kind of command/request API that you wish, it's all a matter of how you write the sketch that decodes it and responds to it.

Isn't there an index of yun-related tutorials, or a single step-by-step Yun guide?

There are some pages that give some good overall information on the Yun, but you won't find a single step-by-step guide because there are so many things that you can do with the Yun, and so many ways to do it. For example, even your simple question about sending commands to a sketch can be done many ways. But good initial reading is:

In addition, Jesse has started to collect some useful information here: 3rd Party Projects for the Yun

Till now what I understood and what I miss is:

  • Create a suitable webpage (how??)

That's a really broad question, and out of scope for this forum. You can write raw HTML pages in any text editor, you can use HTML editors that simplify the process, or many word processors (such as Microsoft Office Word) lets you save documents as HTML pages (although the results are often messy.) And what you should put in the web page is totally up to you. Searching the web on creating web files will inundate you with information. But discussing it here is overly broad and off-topic.

  • Setup a [yunname].local/ address (how?? is it done by Arduino IDE installer or what? Can I setup such address by installing just the ZIP version of the IDE?)

You give your Yun a name during it's initial configuration. The configuration process is spelled out here. The name you enter in the "YUN NAME" field becomes the network name of the device. For example, if you enter "MyYun" in that field, the resulting network name will be MyYun.local.

- Can I connect by WiFi to a Yun not connected to internet?

Yes. Of course, to communicate with the Yun, it needs to be connected to a network, but that network does not necessarily need to be connected to the Internet. That being said, certain functionality will need an Internet connection, such as running opkg on the Linux side to install software, and accessing remote services like Temboo or cloud data servers.

2

So, how do you implement web services with the Yun?

There are several ways to accomplish this, the best method depends on your individual goals. In the following discussion, it is not my intent to exhaustively list every option, I just want to touch on the simplest or most common way of implementing a task. There are usually more advanced (and more powerful and flexible) ways of accomplishing a goal, but I'll leave them alone for now.)

Serving up static web pages: If you insert an SD card that has a folder named "arduino" in the root directory, the Yun will serve up any pages that are stored in the "www" folder inside of the "arduino" folder. For example, suppose the SD card has a file named [color=blue]/arduino/www/mypage.html[/color]: That file could be accessed using the URL [color=blue]http://arduino.local/sd/mypage.html[/color]. You can also create subfolders off of www: the file [color=blue]/arduino/www/test/index.html[/color] would be accessed using the URL [color=blue]http://arduino.local/sd/test/index.html[/color]. If just a folder name is given, the Yun will assume you want a file named [color=blue]index.html[/color], so that last example file could also be accessed using [color=blue]http://arduino.local/sd/test[/color]

Serving up dynamic web pages: The web server embedded in the Yun can generate dynamic web pages using a CGI mechanism: basically, you write a script in perl, php, Python, or just about any language that generates the page to be displayed. An example of Python CGI

REST API: You can define a simple REST API to send commands and retrieve status. This was discussed above, and is also covered by the Bridge Example. These calls are synchronous: the process making the request must wait until the sketch receives the and processes the request before the calling process can continue. This can also be expanded to return more complex data, and not just a simple pin state response. For example, the TemperatureWebPanel Example uses a simple request, and gets a detailed response back. While the tutorial goes into the detail about the sketch, it doesn't include the actual web page that fetches the data. The web page uses the zepto JavaScript library, which is essentially a variant of jquery. The web page that fetches and displays the information from the sketch is:

<!DOCTYPE html>
<html>
    <head>
    <script type="text/javascript" src="zepto.min.js"></script>
      <script type="text/javascript">
        function refresh() {
            $('#content').load('/arduino/temperature');
        }
</script>

    </head>
    <body onload="setInterval(refresh, 2000);">
        <span id="content">0</span>
    </body>
</html>

Bridge Key/Value Data: There is a data storage mechanism in the Bridge. These values can be thought of as shared variables that can be read and written by the sketch, by linux programs, or by remote systems making web requests. This is an asynchronous method: values are written as needed, and can be read at any time. String values can be saved and accessed using names. When a value is written using put(), it overwrites an existing old value. When a value is read using get(), it reads the last put value. Note that this really does work like a shared variable: there is no notification to the sketch or Linux code that a value was written, the code must keep looking for changes. Also, just like variables, there is no way to tell that a value was written if the new value is the same as the old value. Because of this, the mechanism is best for reporting status, or for accepting commands where the latest value is important, and it's not important to know exactly when a new value was set (for example, if you are implementing a thermostat, it's important to know the current desired temperature, but not as important to know that the same desired temperature was set twice in a row. There is no way to delete a value once written, but it can be set to a blank value. An example of reading a bridge value and updating a web page based on the response IS HERE.

  • A sketch can set a value using Bridge.put(), and can read a value using Bridge.get().
  • A remote computer can put a value using a URL of the form: [color=blue]http://arduino.local/data/put/key/value[/color], which will store the string "value" under the name "key"
  • A remote computer can get a value using a URL of the form: [color=blue]http://arduino.local/data/get/key[/color], which will read the value named "key" and return a JSON response with the value.
  • A remote computer can get all values using a URL of the form: [color=blue]http://arduino.local/data/get[/color], which will return a JSON response containing all of the currently defined values.

Mailbox: Commands can be sent to the Yun using the Mailbox message queue. Unlike the bridge key/value pairs, this is an actual message queue, and all messages will be received by the Yun, even if the same message is sent twice in a row. This is an asynchronous method: the remote process can send the message to the mailbox, and will return as soon as the message is sent, then the sketch can process the message later when it gets around to it. MailboxReadMessage Example.

3

jumpjack:
I created this page, would it work? (I did not yet purchase a Yun
But how do I pass a parameter from sketch to web page?

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
       

Acceso

Spento

> ```

Your usage of window.open could theoretically work, in that it should make the request and display the results, but it's unlikely that you really want to open (and then have to close) a window with each button press. In addition, it does not let you programmatically process the response from the request, which is why you don't know how to handle: // StatoLed = ? ? ? ; // DEBUG

The basic idea is that you want to make an AJAX style web request from your page. The simplest way to get that status value is probably to use getJSON as described here. The same technique could be used to set values as well, with different handling of the response.

4

ShapeShifter:

jumpjack:
Till now what I understood and what I miss is:

  • Create a suitable webpage (how??)

That's a really broad question, and out of scope for this forum. You can write raw HTML pages in any text editor, you can use HTML editors [...]

Well, I didn't mean such a generic question! :slight_smile:
By "suitable" I meant "capable of sending/receiving data to/from Arduino".

ShapeShifter:

jumpjack:

  • Setup a [yunname].local/ address (how?? is it done by Arduino IDE installer or what? Can I setup such address by installing just the ZIP version of the IDE?)

You give your Yun a name during it's initial configuration. The configuration process[...]

I meant: how does the OS map the IP to the human-readable name? I think unpacking a zip file is not enough to make such an association, and Arduino IDE can also be "installed" by just unpacking a zip.

ShapeShifter:

jumpjack:

  • Can I connect by WiFi to a Yun not connected to internet?

Yes. Of course, to communicate with the Yun, it needs to be connected to a network, but that network does not necessarily need to be connected to the Internet.

Thanks, that's a very good and important information: what I'm attempting to do is turning on a light placed in the middle of nowhere in the country (read: no GSM, no Edge, no 3G, no cabled/wireless network at all) by connecting my phone to the board through wifi.

I had to read and mix-up several pages and your replies to figure out how actually a Yun webserver must be configured, but I think I eventually got it! Thanks a lot.

I hope this step-by-step procedure is correct:

Let's assume you're using Windows, C:\ is the hard disk where you installed Arduino IDE and E:\ is your SD card.

  1. Create a folder structure named E:\arduino\www into root folder of the microSD card;

  2. Create a sketch like this one:

#include <Bridge.h>
#include <YunServer.h>
#include <YunClient.h> 
 
YunServer server;
 
void setup() {
  Bridge.begin(); // Enable connection between Arduino and Linux
  server.listenOnLocalhost(); // Listen only local connections [color=red][b](why just local ones??)[/b][/color]
  server.begin(); // Start Linux web server, so when user accesses http://arduino.local/arduino/xxxxxx or http://192.168.x.x/arduino/xxxxxx address, Linux server Linux sends xxxxxx to Arduino sketch.
}
 
void loop() {
  YunClient client = server.accept(); // Create client, which receives string xxxxxx .
 
  if (client) {
    String command = client.readString();
    command.trim(); // Now object  “command” holds xxxxxx string received from web page
    if (command == "temperature") {
      int val = analogRead(A1); // Access Arduino hardware to read a pin value
      client.print(val); // Send value in plain text to Server which sends it to browser.
    }
    client.stop();
  }
 
  delay(50);
}
  1. Select Sketch->Show Sketch Folder from IDE menu: you'll see the sketch folder on your hard disk;
  2. Create a "www" subfolder into sketch folder, named for example "mysketch", so you'll end with a C:\arduinoIDEpath\mysketch\www structure; all files in the www folder will get uploaded to SD into E:\arduino\www\mysketch folder when you upload the sketch using WiFi or ethernet [to be confirmed](if you use USB to upload sketch, you have to upload web page manually using FTP).
  3. Move SD card into Arduino Yun board; Linux webserver will see the E:\arduino\www\mysketch folder as /mnt/sd/arduino/www/mysketch, and the folder will be accessible by browser as http://boardname.local/sd/mysketch or http://192.168.x.x/sd/mysketch (in other words, E:\arduino\www maps to http://192.168.x.x/sd)
  4. Configure Yun wifi network (section "Configuring the onboard WiFi" in this page) to be accessible by PCs, phones and tablets;
  5. Create an HTML page which allows intercommunication between Arduino world and Linux world:
<html>
 <head>
 <script type="text/javascript" src="zepto.min.js"></script>
  <script type="text/javascript">
   function readData() {
   $('#content').load('/arduino/xxxxxx'); // Command sent to sketch: xxxxxx ; result stored into "content" object defined in HTML body.   }
</script>

 </head>
 <body>
 <input type="submit" onclick="readData();" id="bnt_start" value = "GO!">
 <input type="text" id="content" value = "[data will be stored here]">             
 </body>
</html>

7a. Web pages send data to sketch by means of URLs formed like http://arduino.local/arduino/**xxxxxx** or http://192.168.x.x/ arduino/xxxxxx , where xxxxxx is what sketch actually sees;
7b. Web page receives from sketch data that sketch sends by means of client.println() statement, which sends plain text to the server and hence to the browser; to send an HTML page to the browser, you must use proper HTTP header:

client.println("Status: 200");
client.println("Content-type: text/html");
client.println(); // riga vuota obbligatoria
client.println("<html><body>Valore ricevuto:" + val + "</body></html>") // Pagina web da visualizzare.
  1. Now you have all you need:
  • a sketch
  • a web page
  • a properly formatted SD card
  • a properly configured Yun (wifi ok, ethernet ok, power ok, SD card inserted)

From now on, you can edit your sketch in Arduino IDE and edit your web page in your favourite HTML editor; once you're done, you upload both from IDE in one single step if you're using wifi; if you are using USB and ethernet, you'll upload sketch using IDE and the web page using an FTP manager which supports SCP protocol (like FileZilla or WinSCP), which can access the the SD card over ethernet

URLs for communication from web page to arduino:

Commands to communicate from sketch to web page:

  • client.print(val); - Send value to Server
  • bridge.put(var, val); - Store "val" value into "var" variable of shared memory
  • variable = bridge.get(var, buffer, buffer_length); - Read "var" variable contents from shared memory

I didn't yet purchase a Yun so I can't check if above statements are correct.
Now that I know it can work without internet I'm going to buy it and do some tests.

UPDATE

First test:
I can't see the Yun IP address in the IDE menu if I connect it over ethernet rather than wifi; but I can access files on SD card using 192.168.x.x/sd address I'll do more tests, as I do not have wifi on the old desktop PC I'm using...

5

jumpjack:
Well, I didn't mean such a generic question! :slight_smile:

Hey, you never know... You did post in a "first time user" thread with some basic questions, so I figured it was best to not make any assumptions at all. :wink:

By "suitable" I meant "capable of sending/receiving data to/from Arduino".

I hope that was answered by the rest of the post.

I meant: how does the OS map the IP to the human-readable name?

It is done using mDNS. The Yun sends out broadcasts advertising itself, and the mDNS software in your computer receives that and gets the name from there. Your computer will need an mDNS client, and not all come with one. Windows does not normally have it, but it can be installed as Apple's Bonjour service, which can be installed separately, or is automatically installed if you have iTunes, QuickTime, or some other Apple software products installed. Macs and iOS devices generally have the service installed automatically. Linux needs a client to be installed, I think it's avahi. As far as I know, an mDNS client is not available for Android.

what I'm attempting to do is turning on a light placed in the middle of nowhere in the country (read: no GSM, no Edge, no 3G, no cabled/wireless network at all) by connecting my phone to the board through wifi.

That should work. If there is no other network around, the Yun can act as an access point and your phone can connect to that.

I had to read and mix-up several pages and your replies to figure out how actually a Yun webserver must be configured, but I think I eventually got it! Thanks a lot.

I hope this step-by-step procedure is correct:

You have a pretty good grasp of it. The biggest issues I see is one of order: you can't do things in the order you specify. For example, you talk about uploading a sketch over WiFi, then putting the card in the Yun, then setting up the Wifi network. You need to reverse that order.

I won't duplicate all of the information in your steps, but I will put them in order. (I'm keeping your step numbers for easy reference.)

  1. Configure Yun wifi network

This is pretty much the first step, as most other steps will depend upon it.

  1. Create a folder structure named E:\arduino\www into root folder of the microSD card;

Yes, a variety of the things that the Yun does to simplify things requires that the root folder of the SD card contains a \arduino\ folder. This is the trigger that causes the Yun to mount the SD card as /mnt/sd, and allows the web service features to work. Normally, the SD card is mounted as /mnt/sda1, but if it contains the /arduino/ folder, the Yun will create a link so that /mnt/sd will also reference the card.

You can create this folder on your PC, or on the Yun by connecting to the command line via SSH. If you do it on the Yun, you will then have to reboot the Yun, or remove/insert the card for the special folder to be recognized, and for the /mnt/sd link to be created.

  1. Move SD card into Arduino Yun board

You will, of course, need to do this early in the process, certainly before you try to upload the sketch.

  1. Create a sketch like this one

Yes, that is about the bare minimum for catching a REST request. This step talks about uploading the sketch over WiFi, but it's too early for that step.

  1. Select Sketch->Show Sketch Folder from IDE menu
  2. Create a "www" subfolder into sketch folder
  3. Create an HTML page

At this point, you've got most of the required files on your computer. In the case of the web page you list, it also references the zepto.min.js file: you will also have to put this file in the www folder along with your web page. Also, any other files (like images?) that your page references will need to be included. Now is the time to upload the sketch over WiFi. This will not only program the sketch into the '32U4 processor, but it will also copy all of the files from your local computers "...<sketchname>\www" folder to the Yun's "/mnt/sd/arduino/www/" folder. At this point, you should be up and running.

There is another way to set it up: you can manually copy the required web files (html, js, etc) to the SD card /arduino/www folder. You can do this by plugging the card into your computer, or over the network using SCCP or some other file sharing protocol. If you put these support files in place manually, you now have the option of loading your sketch over USB, if desired.

URLs for communication from web page to arduino:

Commands to communicate from sketch to web page:

  • client.print(val); - Send value to Server
  • bridge.put(var, val); - Store "val" value into "var" variable of shared memory
  • variable = bridge.get(var, buffer, buffer_length); - Read "var" variable contents from shared memory

Good summary! 8)

There is one more question from your original post that I realize I never addressed:

jumpjack:
Additionally, I don't understand when the page talks about "REST" (I know POST and GET) and the required password...

While I talked about REST, I forgot the password part. Back when the Yun is originally configured (where the name is being set) you also set a password. This is the password that you will use for:

  • logging into the Yun's configuration web pages
  • logging into an SSH session with the Yun
  • uploading a sketch over WiFi (or Ethernet)
  • submitting a REST API request
    That last one is the one that is often confusing. When sending a REST API web request to the Yun, you will need to use web standard Basic Authentication with a username of "root" and this password in order to gain access. The alternative is to configure an "open" REST API by checking the appropriate radio button on the Yun's initial configuration page. While the other fields on that page require you to save and restart before they take effect, the "Open" and "With Password" settings take effect as soon as you click them. The "With Password" option is the factory default.
6

@ShapeShifter - full marks for an excellent presentation. I will book mark it.

I hope this is not off-topic, but I can't actually find what is the name of the web-server program that runs on the Yun.

...R

7

Robin2:
@ShapeShifter - full marks for an excellent presentation. I will book mark it.

Thank you! Now, you need to add some discussion about how to do it all more efficiently in Linux and not involve the sketch! 8)

I hope this is not off-topic, but I can't actually find what is the name of the web-server program that runs on the Yun.

uHTTPd

8

ShapeShifter:
Thank you! Now, you need to add some discussion about how to do it all more efficiently in Linux and not involve the sketch! 8)
uHTTPd

Thanks for the link.

I am sending a PM about the other part so as not to clutter this discussion.

...R

9

@jumpjack,

the answers given by ShapeShifter were spot on as always. However, I'm going to make three (3) minor clarifications for future readers of this thread.

ShapeShifter:

I meant: how does the OS map the IP to the human-readable name?

I hope that was answered by the rest of the post.
It is done using mDNS. The Yun sends out broadcasts advertising itself, and the mDNS software in your computer receives that and gets the name from there. Your computer will need an mDNS client, and not all come with one. Windows does not normally have it, but it can be installed as Apple's Bonjour service, which can be installed separately, or is automatically installed if you have iTunes, QuickTime, or some other Apple software products installed. Macs and iOS devices generally have the service installed automatically. Linux needs a client to be installed, I think it's avahi. As far as I know, an mDNS client is not available for Android.

#1) Actually, the resolution of (domain) Names to (IP) Number is handled by DNS. This goes for all public and private Name-to-IP-Number services. However, DNS has corner cases that are covered by auxillary services; mDNS being one, another being DHCP.

The complete answer is beyond the scope of this forum, so I encourage you to read more as you can, and ask questions.

#2) On the Android side, there is Network Service Discovery, which appears to be similar, if not the same. I have not used it so I have no details.

URLs for communication from web page to arduino:

Commands to communicate from sketch to web page:

  • client.print(val); - Send value to Server
  • bridge.put(var, val); - Store "val" value into "var" variable of shared memory
  • variable = bridge.get(var, buffer, buffer_length); - Read "var" variable contents from shared memory

#3) Everywhere you say "shared memory" exchange that with "cached memory". The "cached memory" is stored in the /tmp directory of the Linux machine. Under many circumstances I would not make this addition, but Linux does have a very specific storage system called shared memory and I did not want there to be any confusion.

Again to be clear, the Bridge library and process uses "cached memory" not shared memory.

Jesse

10

Jesse,thanks for the clarifications. I was, of course, trying to keep it simple and use easily understood metaphors. I fear going into that level of detail might be confusing to beginners?

jessemonroy650:
#2) On the Android side, there is Network Service Discovery, which appears to be similar, if not the same. I have not used it so I have no details.

This looks like a mechanism that can be embedded into applications so that one app can find another one, in a sort of peer-to-peer basis. It looks like a set of APIs that can be embedded into each application that needs it. It does not appear to be a general purpose advertisement scheme like mDNS. To make it work for The Yun, it looks like the advertising code would have to be added to the Yun, and the discovery code added to the IDE, which probably isn't going to happen. And even if that were implemented in the IDE, it still wouldn't help trying to access the Yun from a web browser or an application on another computer.

Android really needs some sort of mDNS client, I don't understand why there isn't one. I've been working on a product with an embedded WiFi module and web server, and the project manager is annoyed that all the computers, and all of the iOS phones and tablets can access it by name, yet she must use an IP address on her Android phone. It is, of course, the same issue with trying to access a Yun by name.

11

Hello,

Thanks for those explanations !

I have one question :
I don't understand how i can send information from the linux part to the sketch without using urls like "http://192.168.x.x/" or "http://myardui.arduino/".

I would like to have a job on the linux part which is going to ask a webserver if a variable is true or false and then send the result to the sketch.

It's going to work fine at home with "http://192.168.x.x/" or "http://myardui.arduino/". But then if I switch networks, the arduino's ip is going to change and my program will no longer work.. No?

What do I miss?

12

qdurand:
I have one question :
I don't understand how i can send information from the linux part to the sketch without using urls like "http://192.168.x.x/" or "http://myardui.arduino/".

I would like to have a job on the linux part which is going to ask a webserver if a variable is true or false and then send the result to the sketch.

Having the Linux side of the Yun communicate to the sketch side using a web service is a bit of overkill, and as you note it can have problems if you are not in control of the network conditions.

This thread is basically about how web services can communicate with the Yun. In your case, you want to do the opposite: have your Yun communicate with a web service and have the results influence the sketch.

You can have your sketch directly access the remote web service using the HttpClient class in the Bridge Library: HTTP Client Tutorial

That's the "easy" way, but it's not the most efficient or powerful way, and I think it puts too much load on the already limited capabilities of the AVR processor. The Linux side has a much more powerful processor with much more memory - I think it makes more sense to have it make the web request (probably using curl?) and then pass the information up to the sketch, exactly as you propose.

The way I like to organize Yun projects is to have the Linux side do as much of the work as possible, and leave the sketch to do the parts that it does best, which is the control of the shield pins and external hardware. There are a few ways to do this, but I like to use the Process class of the Bridge Library: I write a program that runs under Linux, using either a shell script, a Python script, or any other method.'

I usually write a Python script: it simply prints out anything that should go to the sketch to standard output, just as it would print to the screen. And it accepts data from the sketch by reading standard input, just as it would read from the keyboard. I test out the Python script by running it from the command line, simulating the input from the sketch by typing on the keyboard, and watching the output to the sketch by seeing what it prints on the screen.

Once the Linux process is working, then I concentrate on the sketch. In the sketch, I use an instance of the Process class that starts and runs the Linux process. When the Linux process is started like this, anything that it outputs is sent back to the Process object, and that output can be read by the sketch using the same .available() and read() functions that you would use on a serial port. The Process object also has the same print() and println() functions that a serial port would have, and anything you print to the Process object gets sent to the Linux script's standard input.

The Process Tutorial shows running a process and getting the output from it. Unfortunately, it doesn't show sending anything to the Linux process. While the tutorial shows running existing Linux commands, if you write your own script on the Linux side, there is no reason you can't call it from a Process object.

There is quite some discussion on this forum about the Process object, which covers a lot more details than this short post.