How to use persistent connections?

I went to the website below to test my REST API & AJAX design.

http://www.webpagetest.org/

And here are the suggestions:

Use persistent connections (keep alive): 14/100

FAILED - http://175.181.128.89/sd/
FAILED - http://175.181.128.89/sd/off.jpg
FAILED - http://175.181.128.89/arduino/status/1
FAILED - http://175.181.128.89/arduino/status/1
FAILED - http://175.181.128.89/arduino/status/1

Is it possible to keep alive the client request on Arduino YUN?
How may i set up the system to meet the requirement?

<html>

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> 

<title>Arduino Yun I/O Demo</title>

<script type="text/javascript">


window.onload=Pinstatus;

function Pinstatus(){

morestatus();

}

function morestatus(){

setTimeout(morestatus, 800);

//document.getElementById("description").innerHTML = "Processing Status";

server = "/arduino/status/1";

request = new XMLHttpRequest();

request.onreadystatechange = updateasyncstatus;

request.open("GET", server, true);

request.send(null);

}

function updateasyncstatus(){

if ((request.readyState == 4) && (request.status == 200))

{

result = request.responseText;

//document.getElementById("description").innerHTML = result;

fullset = result.split("#");

//document.getElementById("description").innerHTML = fullset;


for(var i = 1, arrayLength=fullset.length; i < arrayLength; i++){

PinPair = fullset[i];

singleset = PinPair.split("=");

PN = singleset[0];

Pinstatus = singleset[1];

if (PN > 10)

{

ActNum = "action" + PN;

ImgNum = "image" + PN;


if (Pinstatus == 0)

{

PinAct = "1";

image = "off.jpg";

}

else

{

PinAct = "0";

image = "on.jpg";

}


document.getElementById(ActNum).value = PinAct;

document.getElementById(ImgNum).src = image;

}


}

}

}

function sendbutton(Pin,action){

//document.getElementById("description").innerHTML = "Processing Button Click";

server = "/arduino/digital/" + Pin + "/" + action;

request = new XMLHttpRequest();

request.onreadystatechange = updateasyncbutton;

request.open("GET", server, true);

request.send(null);

}

function updateasyncbutton(){

if ((request.readyState == 4) && (request.status == 200))

{

result = request.responseText;

//document.getElementById("description").innerHTML = result;

singleset = result.split(",");

PinType = singleset[0];

PinNum = singleset[1];

Pinstatus = singleset[2];

ActNum = "action" + PinNum;

ImgNum = "image" + PinNum;

if (Pinstatus == 0)

{

PinAct = "1";

image = "off.jpg";

}

else

{

PinAct = "0";

image = "on.jpg";

}

document.getElementById(ActNum).value = PinAct;

document.getElementById(ImgNum).src = image;

//ocument.getElementById("description").innerHTML = result;

}

}

</script>

</head>

<font face="Arial">

<table name="Table" border="1" cellpadding="6">

<tr> <th align="center" colspan="6" >控制面板</th></tr>

<tr>

<td align="center">

air conditioner 1




<input type="hidden" name="pin" value="12" id="pin12" />

<input type="hidden" name="action" value="0" id="action12" />

<img src="off.jpg" width="50" id="image12" onclick="sendbutton(document.getElementById('pin12').value,document.getElementById('action12').value);"/>

</td>

<td align="center">

heater




<input type="hidden" name="pin" value="13" id="pin13" />

<input type="hidden" name="action" value="0" id="action13" />

<img src="off.jpg" width="50" id="image13" onclick="sendbutton(document.getElementById('pin13').value,document.getElementById('action13').value);"/>

</td>

<td align="center">

air conditioner 2




<input type="hidden" name="pin" value="11" id="pin11" />

<input type="hidden" name="action" value="0" id="action11" />

<img src="off.jpg" width="50" id="image11" onclick="sendbutton(document.getElementById('pin11').value,document.getElementById('action11').value);"/>

</td>

</tr>


</table>


<p id="description">  </p>

</font>

</html>

https://varvy.com/pagespeed/keep-alive.html

HTML5 WebSocket represents the first major upgrade in the history of web communications. Before WebSocket, all communication between web clients and servers relied only on HTTP. Now, dynamic data can flow freely over WebSocket connections that are persistent (always on), full duplex (simultaneously bi-directional) and blazingly fast.

The python websocket client example:

What are you trying to accomplish with this testing? That site seems to be a way to test a web site's overall speed -- you do realize that the Yun is not a high performance server, right? That's especially true of a REST API implemented in a sketch - it's designed for ease of programming, not for performance. If it's speed you're after, there are better ways to do it, like implementing it all on the Linux side.

But on to your immediate question:

tim9510019:
Is it possible to keep alive the client request on Arduino YUN?
How may i set up the system to meet the requirement?

First off, it's suggestion, not a requirement.

The Yun uses the uhttpd web server daemon, so the Apache/NGINIX/Lightspeed suggestions in your final link won't help you (but you probably already figured that out.) The configuration options for uhttpd are described here: Web Server Configuration (uHTTPd) [Old OpenWrt Wiki]. There is a configuration command for http_keepalive, which I notice is not set in the Yun's default configuration file. The documentation seems to indicate that there is a default value of 20 seconds, but there is also a warning about some bugs with that command.

Of bigger concern to me are the failure messages - Is that just a list of the pages that it thinks would work better with keep-alive? Or are there actual failures to load those files?

You may see some improvement in using keep-alive, but I'll bet it won't be as dramatic as you hope. The REST API as implemented by a sketch is not a high performance system. Like I said, it's designed to be easy to code. To accomplish that, there's a lot of overhead going on behind the scenes. In highly simplified terms: The uhttpd web server running on the Linux side accepts the incoming http request. It sees that the first token in the line is /arduino/ so it opens a socket to talk to the sketch. Using the serial port that is between the Linux and sketch processors, opening this socket triggers the YunServer object to open its end of the socket and return a YunClient object. Of course, this won't happen until the sketch gets around to calling the YunServer's accept() function. The Linux side then send the remainder of the URL (the part after /arduino/) to the YunClient, and that data also has to go over the serial port between the processors. The sketch code then parses that request and generates a response, which it writes to the YunClient object. That data then gets sent over the serial port to the Linux processor, which packs it up as a proper HTTP response, and finally sends the data out.

That's a lot of processing, involving processing overhead on the Linux processor, a lot of communications over the serial port, latency in the sketch loop() function getting around to calling YunServer.accept(), creating the YunClient object and parsing the data, sending the data back, and so on. The Linux processor is not a high speed computer, but in comparison, the sketch processor, and the serial port are much slower.

That's all a long-winded way of saying that you may increase the response time slightly by using keep-alive, but the overhead of going up to the sketch and back down on each REST API request is going to far outweigh any savings.

If you want to really speed things up, don't worry so much about the minor tuning parameters like keep-alive, and work on eliminating the real bottlenecks like the sketch and the serial communications. It's more work, but much more efficient to implement as much as possible on the Linux side of the Yun.

After I used the websocket technology, the response time was reduced a lot.

Original Httpd AJAX design always creates a new connection between client and host.
And it detects the connection status whenever a comment or action is made.
Even worse, it can't deal with multi-users' request because the Yun resource is limited and it's waste of time to send and check connection status every time.

Moreover, because my application detected the pin status all the time, some problems occur:

When a client is connected, it refreshes status every 300ms.
When two clients are connected, it refreshes status every 2 secs and cause a high latency after a while.
I observe the phenomenon through Firefox web analyzer.

The websocket server is set on port 8080 by arduino yun with micro SD expension.
Here are the instructions I followed:

http://tavendo.com/blog/post/arduino-yun-with-autobahn/

The Cons of this tutorial is that it abandons original bridge function, so Bridge.begin() and REST API can't be used. The python code serial2ws.py connect Arduino MCU serial port with websocket. So the control can be achieved.

But its python code is too weird for me to understand. Some function call from nowhere but it still works. Maybe it's remote function call, but i don't know where it comes from. Could somebody give me some explanations? I will appreciate that.

The two mysterious functions are as below:

u"com.myapp.mcu.on_analog_value"
u"com.myapp.mcu.control_led"

Here is his python code, serial2ws.py:

###############################################################################
#
# The MIT License (MIT)
#
# Copyright (c) Tavendo GmbH
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
###############################################################################

from twisted.internet.defer import inlineCallbacks
from twisted.internet.serialport import SerialPort
from twisted.protocols.basic import LineReceiver

from autobahn.twisted.wamp import ApplicationSession


class McuProtocol(LineReceiver):

    """
    MCU serial communication protocol.
    """

    # need a reference to our WS-MCU gateway factory to dispatch PubSub events
    def __init__(self, session):
        self.session = session

    def connectionMade(self):
        print('Serial port connected.')

    def lineReceived(self, line):
        print("Serial RX: {0}".format(line))

        try:
            # parse data received from MCU
            data = [int(x) for x in line.split()]
        except ValueError:
            print('Unable to parse value {0}'.format(line))
        else:
            # create payload for WAMP event
            payload = {u'id': data[0], u'value': data[1]}

            # publish WAMP event to all subscribers on topic
            self.session.publish(u"com.myapp.mcu.on_analog_value", payload)

    def controlLed(self, turnOn):
        """
        This method is exported as RPC and can be called by connected clients
        """
        if turnOn:
            payload = b'1'
        else:
            payload = b'0'
        print("Serial TX: {0}".format(payload))
        self.transport.write(payload)


class McuComponent(ApplicationSession):

    """
    MCU WAMP application component.
    """

    @inlineCallbacks
    def onJoin(self, details):
        print("MyComponent ready! Configuration: {}".format(self.config.extra))

        port = self.config.extra['port']
        baudrate = self.config.extra['baudrate']

        serialProtocol = McuProtocol(self)

        print('About to open serial port {0} [{1} baud] ..'.format(port, baudrate))
        try:
            serialPort = SerialPort(serialProtocol, port, reactor, baudrate=baudrate)
        except Exception as e:
            print('Could not open serial port: {0}'.format(e))
            self.leave()
        else:
            yield self.register(serialProtocol.controlLed, u"com.myapp.mcu.control_led")


if __name__ == '__main__':

    import sys
    import argparse

    # parse command line arguments
    ##
    parser = argparse.ArgumentParser()

    parser.add_argument("--baudrate", type=int, default=9600, choices=[300, 1200, 2400, 4800, 9600, 19200, 57600, 115200],
                        help='Serial port baudrate.')

    parser.add_argument("--port", type=str, default='2',
                        help='Serial port to use (e.g. 3 for a COM port on Windows, /dev/ttyATH0 for Arduino Yun, /dev/ttyACM0 for Serial-over-USB on RaspberryPi.')

    parser.add_argument("--web", type=int, default=8000,
                        help='Web port to use for embedded Web server. Use 0 to disable.')

    parser.add_argument("--router", type=str, default=None,
                        help='If given, connect to this WAMP router. Else run an embedded router on 8080.')

    args = parser.parse_args()

    try:
        # on Windows, we need port to be an integer
        args.port = int(args.port)
    except ValueError:
        pass

    from twisted.python import log
    log.startLogging(sys.stdout)

    # import Twisted reactor
    if sys.platform == 'win32':
        # on windows, we need to use the following reactor for serial support
        # http://twistedmatrix.com/trac/ticket/3802
        from twisted.internet import win32eventreactor
        win32eventreactor.install()

    from twisted.internet import reactor
    print("Using Twisted reactor {0}".format(reactor.__class__))

    # create embedded web server for static files
    if args.web:
        from twisted.web.server import Site
        from twisted.web.static import File
        reactor.listenTCP(args.web, Site(File(".")))

    # run WAMP application component
    from autobahn.twisted.wamp import ApplicationRunner
    router = args.router or u'ws://127.0.0.1:8080'

    runner = ApplicationRunner(router, u"realm1",
                               extra={'port': args.port, 'baudrate': args.baudrate},
                               standalone=not args.router)

    # start the component and the Twisted reactor ..
    runner.run(McuComponent)

The example transfers single word to serial port to control arduino pins. (control flow with ASCII code)
I modified his example, and i send upper case letters for 'ON', and lower case letters for 'OFF'.

Here is my demo web:
http://175.181.124.83:8000

Finally, i have to say. I want to keep my application easy but powerful.
And I love to use original bridge functions, so i am seeking an alternative solution now.

Recently i found node.js examples, but i don't know how to download socket.io and serialPort
modules to the folder. The web doesn't focus on the how-to.

Many thanks for help and discussion.

sonnyyu:
HTML5 WebSocket represents the first major upgrade in the history of web communications. Before WebSocket, all communication between web clients and servers relied only on HTTP. Now, dynamic data can flow freely over WebSocket connections that are persistent (always on), full duplex (simultaneously bi-directional) and blazingly fast.

The python websocket client example:

http://forum.arduino.cc/index.php?topic=217316.msg1590152#msg1590152

Dear sonnyyu:
I want to know how to combine this python example with Bridge pin control of arduino yun.
Maybe the question is easy for you, but i just can't figure out.
Could you share more for us to practice and learn?

Thanks.

Now my DEMO web can pass the keep alive test!!!

https://varvy.com/pagespeed/keep-alive.html

Web url:

http://175.181.124.83:8000

ShapeShifter:
If you want to really speed things up, don’t worry so much about the minor tuning parameters like keep-alive, and work on eliminating the real bottlenecks like the sketch and the serial communications. It’s more work, but much more efficient to implement as much as possible on the Linux side of the Yun.

I think you’re right about the whole concept.
Do you have some recommended references dealing with the bottleneck of sketch and the serial communications? And i wanna know how to connect to web like Process class does but without it.
Thanks for your help really!!

tim9510019:
Do you have some recommended references dealing with the bottleneck of sketch and the serial communications?

I try to do as much as possible down on the Linux side, and let the sketch work pretty much as a dumb I/O processor. I handle all of the incoming web requests (including a REST API) in Python using the Bottle framework. I didn’t choose Bottle for any particular reason, other than it was available and the documentation looked like it would be pretty easy to pick up. Other frameworks like Flask or something more advanced would be just as applicable - if you have one you already use, go with that.

I have one system which controls a lighting load according to a schedule, and provides a web interface for control and reporting. The sketch has the minimum amount of logic possible: it periodically receives a 1 or 0 from the Python process to tell it whether the output relay should be on or off, and it periodically measures the current output using a current transformer and analog input, and periodically sends the raw ADC readings down to the Python task.

Another system has a user interface with an RGB backlight 4x20 character LCD display and some LEDs that are under control of the sketch. The sketch includes an incoming data parser that looks for escape sequences: If it sees an escape character, it looks for a series of one or more numbers (as ASCII strings) that are separated by semicolons, until it sees a non-numeric character. It then uses that character as a command, and the previously received numbers as parameters, and processes the command. Anything received outside of an escape sequence goes straight to the LCD screen. I implemented escape sequences such as:

  • C - Clear screen
  • 3;10G - Go to row 3, column 10
  • 255;128;0B - Set backlight to orange (full red, half green, no blue)

In this way, the Python code can control all of the text output and formatting, and also control some additional LEDs and outputs by interleaving the required escape sequences with the text to be displayed.

In all cases, the communications between the sketch and Python are handled with the Bridge Library and Process class objects. The sketch starts one or more (depending on the application) Python scripts, and talks back and forth through the Process object. In this case, it’s important to make sure the Python process is running unbuffered, so that any output from Python is sent to the sketch right away. I do this by including -u in the shebang at the beginning of the python code ([color=blue]#!/usr/bin/python -u[/color]) and then just referencing the Python script in the Process run call. Or you could just use [color=blue]python -u scriptname.py[/color] and set the option directly.

Some of my projects will launch several Python scripts: for example, the lighting controller launches a script that communicates with the sketch, and updates status in SQLite3 and RRDtool databases. There are two SQLite3 databases, one on SD card that contains an activity log and configuration settings, and another in /tmp (a RAM disk) that contains current status and commands.) Another Python process will run the Bottle application script, taking/setting information in the two SQLite3 databases, and a third one periodically checks a dedicated mailbox for incoming emails that allow remote control and status queries. While the main process does the communicating with the sketch, and the other processes don’t send data back directly, they could. In all cases, the sketch checks whether the process object is still running, and if it isn’t, it closes the process and launches it again (good in case there is latent bug that causes the Python script to crash.) I have a writeup of some of the lighting controller features and programming concepts here: http://forum.arduino.cc/index.php?topic=367981.0

The whole philosophy is to do as much as possible on the Linux side, and minimize the data traffic between Linux and the sketch. As a secondary goal, I try to keep the sketch as “dumb” as possible - I find it easier to update and test the Python code and try to minimize the amount of times I have to upload a new sketch

Now, with all that being said, my projects typically have low data rates - I’m not trying to stream lots of data from the sketch. If you are looking for very high bandwidth between the two sides, you are probably better off abandoning the Bridge Library, and manually opening the serial port between the two sides, and writing your own communications protocol.

I hope this gives you some ideas, feel free to ask questions.

And i wanna know how to connect to web like Process class does but without it.

I understand each word in that sentence, yet I can’t really figure out what you’re asking. Can you rephrase it? Perhaps with a specific example?

ShapeShifter:
I understand each word in that sentence, yet I can't really figure out what you're asking. Can you rephrase it? Perhaps with a specific example?

I like your designs and concepts. They really benefit me a lot.
About the question, i'm now using python example code to connect MCU serial port with websocket directly.
And i comment /dev/ttyATH0 in /etc/inittab. So i can't use Bridge related function including REST API and Process class. Previously, i wrote like this as Sonnyyu taught me to output the data from mcu to external database. (PHP GET function)

...
  Process p;               
  p.begin("curl");
  p.addParameter("http://timblog.tk/db/db.php?temperature=" + String(temperature));
...

But now, I can't use the bridge, so I wanna know how i can do the same thing as before.
Do I need another python code to deal with it?
Is it hard to make one?

Beside, i am wondering how i could know which function to be imported to python code.
How can I be familiar with those classes and functions?

Something like this in python:

from twisted.internet.defer import inlineCallbacks
from twisted.internet.serialport import SerialPort
from twisted.protocols.basic import LineReceiver
from autobahn.twisted.wamp import ApplicationSession

I agree with your points of view about how to speed up the adruino application.
That is, let the more powerful Linux system do more complicated things, and leave some simple work (simple data receive and transmit) for the MCU. I also agree with your thought of debugging. It's much more easier to debug Python on console than to update the sketch.

ShapeShifter:
I have a writeup of some of the lighting controller features and programming concepts here: http://forum.arduino.cc/index.php?topic=367981.0

It's very useful!! Bookmarked.
I like the graph that shows current and working state at the same time.
If the chart can be more stylish, it would be better.

tim9510019:
And i comment /dev/ttyATH0 in /etc/inittab. So i can't use Bridge related function including REST API and Process class.

I'm sorry, I must have misunderstood. In your original post, you are using URLs like:

http://175.181.128.89/arduino/status/1

These are by definition using the sketch to implement a REST API. Calls such as these caught by the web server, pre-processed, and sent to the sketch using the Bridge. How are you able to do that if you've bypassed the bridge?

...

Process p;              
 p.begin("curl");
 p.addParameter("http://timblog.tk/db/db.php?temperature=" + String(temperature));
...




But now, I can't use the bridge, so I wanna know how i can do the same thing as before.
Do I need another python code to deal with it?

You will need something on the Linux side to make that call, whether it be Python or another language. Keep in mind that the Process class is just running Linux commands - the curl capability is already available directly on the Linux side (you could type that command on the command line right now) so it's not necessary to reinvent the command, just call it from your Linux process.

Python has libraries to call curl() directly, in which case you'd need to do a minimal translation of the arguments to match the library's calling conventions. Or, Python has the ability to call shell commands directly, so you could just call the existing command directly (easier, but not quite as efficient, but good enough for most purposes.)

Beside, i am wondering how i could know which function to be imported to python code.
How can I be familiar with those classes and functions?

Google is your friend. It's no longer necessary to go to Greece and climb Mt Pernassus to find the Oracle of Delphi - the great Oracle Google knows all, and tells all. There is no substitute for basic research.

That is, let the more powerful Linux system do more complicated things, and leave some simple work (simple data receive and transmit) for the MCU.

In your case, it's made a bit more difficult because you eliminated the Bridge functionality. You can still have serial communications between your Linux process and the sketch. From the sketch's point of view, it's not that much different. Rather than using a Process object, you use Serial1. After that it's the same, as they both derive from the Stream class, so they support the same interface methods.

The Linux side becomes a bit more complicated. You need to use other methods to launch the process (like using rc.local) and you must explicitly open the serial port and send/receive all data through that port. Using the Process object, you would just read/write through standard I/O: this can make debugging easier as you can test the sketch communications from the command line - just launch the process manually from the SSH command line, and any output it sends to the sketch shows up on the terminal, and you can directly type any simulated output from the sketch (the input to the Linux process.)

As you mentioned, disabling the Bridge eliminates a lot of features. It limits you to communications with a single process, and eliminates all of the other Bridge features. For example, the first thing I do in all of my Yun sketches is to use Bridge.put() to put the sketch name, version number, and build date to the Bridge. That way, I can use a web browser to access http://arduino.local/data/get to be able to verify just what's loaded on the Yun. And while most of my sketch/Linux communications are through Process objects, there are times I use Bridge put()/get() or the Mailbox class for special situations. That flexibility is gone without the Bridge. Also, I rely on the Process.running() function to tell me if the process has died, and if so I can restart it. With the Bridge disabled, you have no way of detecting when your process has died, and no way to restart it without manual intervention.

I can see the benefit of disabling the Bridge when you have a high data rate between the sketch and the Linux process - the Bridge does add a certain amount of overhead to be able to do all of the things it does. But by using intelligent design, and minimizing the traffic between the Bridge and Linux, you can still live with the Bridge quite nicely. Personally, I've not yet run into a situation where the benefit of bypassing the Bridge has been worth anywhere near the cost.

tim9510019:
If the chart can be more stylish, it would be better.

They suit my needs, I don't like a lot of chartjunk that distracts from the data being presented. But the tool is capable of a lot more than I use: do a Google image search of "rrdtool graph examples" and you'll see the possibilities. If you want something more, you can always extract the data and use some other tool to generate the graphs.

My original design was based on AJAX design.
It created a connection to REST API address whenever the web received or sent data from/to MCU.
And the trigger condition is base on "readyState ==4" and "status == 200".

http://www.w3schools.com/ajax/ajax_xmlhttprequest_onreadystatechange.asp

I found from the web analyzer that it created so many REST API connections to update current pin status to the web.

With websocket, it created only one connection, but it could send and receive data on this connection all the time.

Thanks for your help.
All the suggestions are noted.
:slight_smile:

Another persistent connection technology:

In May 2011, Google released an open source project for browser-based real-time communication known as WebRTC. This has been followed by ongoing work to standardise the relevant protocols in the IETF and browser APIs in the W3C.

WebRTC is designed for high-performance, high quality communication of video, audio and arbitrary data. It beats WebSocket on speed but not reliability.