I'm working on an Arduino project that responds to commands from the internet. The commands could come at any time of day and the Arduino must get the command instantly (i.e. without polling). I have this working in a controlled environment, with the Arduino set up as a web server and I'm hitting the Arduino from a public Linux server. Everything works just great because I have access to the router settings and can set everything up so that requests can be passed from the internet through this local network to the Arduino directly.
Here's the problem. I need to be able to install this project in places where I won't have access to router settings and where it really needs to be "plug and play." I imagine then that the Arduino must create an outbound connection and keep it open (and reopen it when it closes), waiting for commands from the server. What's the best way to accomplish this?
It has been suggested that I mess around with HTTP keepalive. But I'm wondering how hard it would be to write a custom server to accept the Arduino's TCP connection, keep it open, and write commands to the Arduino when appropriate. Any advice would be appreciated.
I use this. I have managed to keep a connection open for about 8 hours. If the connection is broken, it will quickly re-establish the connection.
Is the Arduino going to be the client? I wrote my own server software for Linux, so that will depend on your skill level to program the server end.
Basically, the client sends a "request" with the data it would normally provide to the server, using a "stop sequence" to signal End Of Transmission (EOT). The server responds with any changes required, and its EOT sequence. Just don't close the connection on either end. Works like a ping-pong game.
I'm working on an Arduino project that responds to commands from the internet. The commands could come at any time of day and the Arduino must get the command instantly (i.e. without polling).
You say this like checking, on every pass through loop, for a new connection take forever. Just how "instantly" do you need this to be? Why?
I need to be able to install this project in places where I won't have access to router settings and where it really needs to be "plug and play."
You seem to expect just to find a network jack to connect to, and be able to bypass all the security that the network/routers/sys-admins impose. Ain't gonna happen.
if you want it to be "plug and play", then you need to find a way to implement UPnP protocol, which is, universal plug and play, exactly what you need. I don't know if anyone has implemented it for arduino yet, but I think it would be a very useful addition to the ethernet library.
If you set up the Arduino as a client, and let it establish the connection, it becomes pretty much plug-and-play. It grabs a localnet ip from a dhcp server, and opens the connection to your Linux server. That eliminates the need to modify the router at all.
I was updating one small packet 15 times per second very comfortably on a localnet. Internet traffic will slow that up a bit if not just localnet.
The advantage of this configuration is that my Linux server will start a new process to service each connection. That means multiple simultaneous connections. The Arduino will only service one persistent connection at a time as a server or client, but as a client, that is all you need.
SurferTim:
I use this. I have managed to keep a connection open for about 8 hours. If the connection is broken, it will quickly re-establish the connection.
Is the Arduino going to be the client? I wrote my own server software for Linux, so that will depend on your skill level to program the server end.
Basically, the client sends a "request" with the data it would normally provide to the server, using a "stop sequence" to signal End Of Transmission (EOT). The server responds with any changes required, and its EOT sequence. Just don't close the connection on either end. Works like a ping-pong game.
OK, thank you. You've got the right idea of what I want to do. The Arduino is the client, using DHCP/DNS it connects to my server on the internet (from behind any firewalls) and then the server pushes the commands as they arrive from the web. Can you lead me to resources about the server software you wrote? I figure I could hack up some sort of telnet server example to get this done.
You say this like checking, on every pass through loop, for a new connection take forever. Just how "instantly" do you need this to be? Why?
I'm starting off with one piece of hardware as the client, but it may turn out to be 30 pieces of hardware all over the country opening a connection to my server. I figure 30 persistent connections makes more sense (and would reduce latency to an absolute minimum) than 30 connections that are opened/closed on a very tight loops.
You seem to expect just to find a network jack to connect to, and be able to bypass all the security that the network/routers/sys-admins impose. Ain't gonna happen.
Actually, in this particular case, I can expect that the locations will have DHCP routers and don't block outgoing connections on any ports.
If you have no control on the routers on each location, you can't expect that any port will be allowed for outgoing connection.
In some case, only web access might be permitted.
You could use an connection over port 80 but you would have to fully respect the HTTP protocol because the firewall might intercept and check validity of the requests (proxy).
So far I've found that using port 443 (HTTPS) is the best port to use for any proprietary usage as proxies are not supposed to be able to intercept and check encrypted content.
Actually, It does not requires you to implement any level of TSL, just use it.
For exemple I've used telnet (pure text protocol) over port 443 and it can goes out from every place where I am using it (hotel hotspot, work , ...)
The Linux server code is the hard part. I ended up with a server that had three threads running. One handled the internet connection, another handled the joystick input, and the last handled the video display and keyboard input on the server. Getting the threads to talk to each other without corrupting data was the challenge, not the comm with the Arduino. That part was easy!
It has none of the thread stuff for the user interface on the server side. You will need to add that to suit your requirements.
I was sending bytes, not characters, that included values from 0 to 255, so a single byte send did not work out. I found the best way to send a stop sequence was to send the data bytes as two bytes. A data byte and a signal byte.
The first byte sent is a data byte, the second byte (signal) sent is zero.
The third byte sent is the second data byte, and the fourth byte (signal) sent is zero.
...and on until...
The next to last byte is a data byte, and the last byte sent (signal) is 255, signalling EOT.
@barbudor: I use ports 8081 to 8088 for projects like this. They have never let me down behind any router.
barbudor:
If you have no control on the routers on each location, you can't expect that any port will be allowed for outgoing connection.
In some case, only web access might be permitted.
You could use an connection over port 80 but you would have to fully respect the HTTP protocol because the firewall might intercept and check validity of the requests (proxy).
So far I've found that using port 443 (HTTPS) is the best port to use for any proprietary usage as proxies are not supposed to be able to intercept and check encrypted content.
Actually, It does not requires you to implement any level of TSL, just use it.
For exemple I've used telnet (pure text protocol) over port 443 and it can goes out from every place where I am using it (hotel hotspot, work , ...)
Sorry, I've been leaving out certain details in order to keep things brief. I know that the locations allow all outbound connections. I just can't go into the location's routers settings and start punching open inbound ports. After some research, I think I'm moving in the direction of telnet right now, so that advice does help in case things change.
The Linux server code is the hard part. I ended up with a server that had three threads running. One handled the internet connection, another handled the joystick input, and the last handled the video display and keyboard input on the server. Getting the threads to talk to each other without corrupting data was the challenge, not the comm with the Arduino. That part was easy!
It has none of the thread stuff for the user interface on the server side. You will need to add that to suit your requirements.
I was sending bytes, not characters, that included values from 0 to 255, so a single byte send did not work out. I found the best way to send a stop sequence was to send the data bytes as two bytes. A data byte and a signal byte.
The first byte sent is a data byte, the second byte (signal) sent is zero.
The third byte sent is the second data byte, and the fourth byte (signal) sent is zero.
...and on until...
The next to last byte is a data byte, and the last byte sent (signal) is 255, signalling EOT.
@barbudor: I use ports 8081 to 8088 for projects like this. They have never let me down behind any router.
Thank you, I'm going to dig into this and see what I can do. I'm also going to try having the Arduino telnet into my server and execute a custom C program (or maybe Python) which will persistently run and pass the commands as characters via stdout. I'm just trying to figure out how to get the web interface to trigger the C program... hmm...
