Web Server Gateway Interface (WSGI) daemon for serial

[ron_sutherland]

The daemon has now been set to allow the origin to be from everywhere, which means Cross-Origin Resource Sharing (CORS) does not need turned off in the browser. The reference I used to do this is over seven years old so if someone has a different idea please suggest.



The files I used to do this are in https://github.com/epccs/RPUpi/tree/e29239e4391a8bb64c23b63f60e7901c8e0ae7e9/WSGIdaemon

[ron_sutherland]

The "client.html" file is a flat stand-alone HTML sourced from the disk with Edge, Chrome, or Firefox. I have modified it to run through a short list of serial port commands and populate some browser elements. I set the daemon to run on an Ubuntu 18.04 machine near my test bench (it has the local network address 192.168.0.7) and ran the stand-alone HTML file in three browsers on a Windows 10 computer.



These are the files I used

https://github.com/epccs/RPUpi/tree/67a1f76fa206edf136d59ef67d7a5b678ad5afd2/WSGIdaemon

the changes form last time are probably the thing to look at

https://github.com/epccs/RPUpi/commit/67a1f76fa206edf136d59ef67d7a5b678ad5afd2

Since I have very little experience with the browser DOM, I suspect my next experiments will be fairly silly. I will just dump the results in Github.

Why is the callback mess done with recursion? Welp, I saw a way to add some more hell to that callback hell, but I'm trying to keep the daemon happy so does that make sense? It wants to do one task at a time, not all at once. The recursion makes sure that each task is done before the next begins, though I doubt it is the best way to do stuff like this.

[Robin2]

I have only had the briefest look at your Python program but it seems to me you are re-inventing the wheel. Aren't you just creating a Python webserver?

...R

[ron_sutherland]

it seems to me you are re-inventing the wheel

hmm...

https://github.com/epccs/RPUpi/blob/67a1f76fa206edf136d59ef67d7a5b678ad5afd2/WSGIdaemon/WSGIdaemon.py#L24

That is not what I call re-inventing, and as further evidence, I am fairly sure I did not learn how to do a web server.

I am not sure where it is going, so it may be a dead end. I looked at bottle and flask frameworks a little but they seem to be unnecessary (they are web servers). I want it to take a CGI style request and turn that into a serial command. An AVR will then take the command and send back a JSON reply, no beautification done by the daemon, just add the headers so the caller is happy, and it will get the pure JSON (I think JS might be able to turn that directly into an object without much fuss but have never tried such things)

Python has batteries included, lots of them. I am connecting Pythons WSGI battery to a serial port and then using another Python battery (the CGI's parse_qs) to piece together query_string elements that match with the commands I have been using on my AVR boards. In short, the daemon spits out the JSON that it was given from the AVR serial, not web pages. It adds headers, but I would not call that a web page. I guess the AVR could spit out HTML pages, and then it would be a web server... wow (but not what I want).

[Robin2]

I am not sure where it is going, so it may be a dead end. I looked at bottle and flask frameworks a little but they seem to be unnecessary (they are web servers).

Bottle is a lot simpler than Flask and has the very nice feature that the entire thing is in a single .py file. It is my "go to" for web related stuff. My initial explorations some years ago was with Ruby on Rails and then Ruby Sinatra which is simpler than Rails. When I switched to Python Bottle seemed a very close analogue to Sinatra.

I have no doubt that Bottle can do a lot of stuff that you don't need, but the way I see it if it can do what I want I will use it - the fact that it can do more would not put me off.

One thing that I discovered that Bottle cannot do "out of the box" is multiple overlapping requests - long polling, when the browser is prepared to wait a long time (seconds, minutes, maybe longer) for a response from the server so as to give the impression to the user of real-time updates when things change at the server end. However I got around that by using a different server engine (if that's the right phrase) in Bottle.

...R

[ron_sutherland]

I have looked at Bottle and Flask some now, but I am not even sure when to bind the constrained resource (e.g. the serial port) in there framework.  So I'm just going to use the core Python WSIG library for now.  

My thinking is: I do not want the serial port to open and close with each browser connection. I need to research the WSGI library and make sure it is single threaded so only one connection is accepted at any time. Sounds like that is what Bottle does, but was that on a host with a single MCUCPU core? That may be more of a problem on an R-Pi3 than an R-PiZero.

When the daemon is running on an R-Pi it will be started with  RC.LOCAL, and I will have to stop it with something like "sudo kill -TERM WSGIdaemon-pid".

https://www.raspberrypi.org/documentation/linux/usage/rc-local.md

Any web browser on my local network can then use the daemon as a Cross-Origin shared resource, so I can serve the client page(s) from any server or as a flat file. I am thinking along the lines of using the flat pages to do simple user interfaces for my boards and projects.

I might use Flask, Bottle, or the Apache server to be the server of the flat web pages but the daemon itself can take calls from any sort of program, perhaps a Python program that periodically gathers data through the daemon to pickle, that way I can gather timed data (e.g. soil moisture levels) while still having the web page based controls at hand.

[Robin2]

I have looked at Bottle and Flask some now, but I am not even sure when to bind the constrained resource (e.g. the serial port) in there framework. 

I don't "bind a serial port in their framework".

I have a single Python program that communicates with an Arduino via serial  (like in this Python - Arduino demo) and which also uses Bottle functions to do web stuff.

...R

[ron_sutherland]

I have a single Python program that communicates with....

Just to be clear my thinking currently is that the daemon is the single program (running 24/7) connected to serial. So I am calling it bound (or should I find another word)  to the serial link. Other programs will communicate with the daemon to get at the serial link (which goes to the microcontroller, actually a multi-drop array of microcontrollers).

[Robin2]

Just to be clear my thinking currently is that the daemon is the single program (running 24/7) connected to serial. So I am calling it bound (or should I find another word)  to the serial link. Other programs will communicate with the daemon to get at the serial link (which goes to the microcontroller, actually a multi-drop array of microcontrollers).

I'm not sure I understand that. I think it means that some "programs" will contact Python through the web interface (in effect those programs will act as web clients) and Python will pass messages to the Arduino.

If my interpretation is correct then that should work fine. For example that is how my Python program presents a Browser based GUI for controlling my model trains. The browser (client) can be on the PC that runs the server or on my smartphone. Or the client could be a program running on another PC or even on an ESP8266.

I only have a vague notion of what a daemon is - I believe it is just a program that runs all the time.

...R

[ron_sutherland]

The idea of a daemon is probably best kept as a vague notion, I sort of think daemons have an uncertainty principle like characteristic where they fall apart or vanish when defined.

I think it means that some "programs" will contact Python through the web interface

Yep, I think many languages can call web interfaces, I recall doing that with Visual Basic (in a distant past) and then trying to deal with XML in the response (ouch).

Python will pass messages to the Arduino

It is just a little more complex, Python needs to look at the query string. So the URL "http://192.168.0.7:8000/?addr=0&cmd=id&q=true" is divided into parts where "?addr=0&cmd=id&q=true" is the query string.  The CGI core Python library has some functions to look inside the WSIG environment and fish out the query string elements. I then can use those elements to craft my serial command for the serial port. But ya the messages get passed to an Arduino is probably the best answer.

If my interpretation is correct then that should work fine.

I mostly like where it is going, so I think the next step is to put this on an R-Pi and see if I can spin it up from there.

[Robin2]

It is just a little more complex, Python needs to look at the query string. So the URL "http://192.168.0.7:8000/?addr=0&cmd=id&q=true" is divided into parts where "?addr=0&cmd=id&q=true" is the query string.  The CGI core Python library has some functions to look inside the WSIG environment and fish out the query string elements.

Bottle has function specifically for that.

This is an extract from an unfinished demo using Bottle

Code: [Select]

@route('/handleForm', method='GET')
def handleForm():
        firstName = request.query.fname
        return template('<b>Hello {{name}}</b>!', name=firstName)
it picks out the element called fname in the query string.

This is the HTML snippet that produces fname

Code: [Select]

           <form id="form1" action="/handleForm">
                <br>
                Enter text: <input type="text" name="fname" value="FirstName">&nbsp &nbsp &nbsp
                <input type="submit" value="Submit"><br><br>
            </form>

In this case the Python program just sends something like Hello Johhny back to the browser. But it could just as easily send it to an Arduino and get a response from the Arduino and pass that to the browser.

...R

[ron_sutherland]

The web service is now returning only the JSON, and can be parsed into a proper JS object. It can then be used to fill the DOM elements as this experiment shows.



This shows the relevant client page updates.

https://github.com/epccs/RPUpi/commit/46f8a8203e538cf4d2f7555dfe3791750d515628

The daemon got some updates with the previous commit.

HTML does not have many controls, and I have not messed with the ones I can find. I see a "meter" tag that looks interesting,  "progress". Are there others?

[ron_sutherland]

A quick look at Python as the client. How difficult will it be to write Python clients that use the web interface gateway?

http://docs.python-requests.org/en/master/

That looks promising. and it is a default install on my Ubuntu computer. 

A quick test and it seems to work.

Code: [Select]

#!/usr/bin/env python3
import requests

r = requests.get('http://192.168.0.7:8000/?addr=1&cmd=id&q=true')
if (r.status_code == 200):
    print( str(r.json()) )
    print( "id.name=" + str(r.json()["id"]["name"]))

WSGIdaemon turns "?addr=1&cmd=id&q=true" into "/1/id?" and sends that to the AVR. The AVR sends back a JSON reply: {'id': {'name': 'KNL', 'desc': 'RPUno (14140^9) Board /w atmega328p', 'avr-gcc': '5.4.0'}}, which the WSGIdaemon sends back, and requests can turn the json right into a dictionary... wow