control without port forwarding

Hi,

I have the following setup: an Arduino with ethernet shield running a simple web server and a page on my 'real' server where the user interface resides. This is a php scripted page doing all the graphic stuff so the Arduino only has to list the status of the used ports and listen to POST commands to change the status of the ports. The Arduino is on port 80 with a fixed IP so I forwarded this IP on my router. Works fine.
But of course it only works in my own home. When I take the device somewhere else I would have to mess with their router to demonstrate it working. Not really an option...
So I'm looking for a way to communicate with the Arduino without the need for portforwarding. This can be done (Teleduino does it, but I don't want to use a third party). I suppose I'll have to setup the Arduino as a client so once it establishes a connection with my server they can communicate without the need for port forwarding, but I have yet to find out how to do so.
Any pointers?

After diving into this topic I now think some form of UDP hole punching is the key to solving this problem.
I think what I need is an Arduino that acts as both a server and a client. To make reading this a bit better I will call the server running on the Arduino server_A and my external Linux server will be called server_L.
The client on the Arduino sends UDP packets through the routers NAT to server_L. The router will translate the Arduino's IP and port to its WAN IP. Server_L then responds to the router which will forward the response to the IP the request came from, so a connection is made between Arduino and Server_L. Then when Server_L sends a request to server_A the routers NAT will route this request to the Arduino. Since the Arduino is also acting as a server it will accept this request and deal with it.
I think in theory this will work, though I have no idea how to implement this in a practical situation.
I'm not even sure if it is possible to setup an Arduino as a server and still have it send UDP/TCP packages at certain intervals to keep the connection alive.
Learning a little bit every day...

You might consider dropping the server component on the arduino and just use tcp. You can write something on your Linux box to manage the conversation with the arduino. Have the arduino initiate the conversation to avoid the issues you mention with the router at the demo location. Then the Linux box can request data/give instructions over the established connection. You can create a kind of keepalive by making the Linux program ask for something frequently even if it isn't required. Then interface the Linux web page to your tcp prog via perl, PHP or your language of choice.

That was my first idea of how it should work. But I could not figure out how to send commands from my Linux box to the Arduino when the Arduino is a client. My understanding is that a client only asks from a server. When the server wants to push something to the client (in this case telling the Arduino to change an output port for example) it requires software on the client side that is not doable on an Arduino. So the Arduino also needs to be a server so it can listen to the Linux box.
Or at least that's how I understand it, but I'm not exactly an expert on these matters...

The BitlashRedisClient example that ships with Bitlash might be a useful source of ideas, even if you don't use the code, which is here: bitlash/BitlashRedisClient.pde at master · billroy/bitlash · GitHub (and more about Bitlash here: http://bitlash.net)

Here's how it works. Redis is an in-memory database server with publish/subscribe capabilities. You set up a Redis instance somewhere (on your web server, or maybe a free one from http://RedisToGo.com). Your Arduino connects to the database and listens for commands (by subscribing to a command channel). Then your PHP web application connects to the database and sends commands to the command channel as needed. Any arduinos connected to the channel receive the commands to be executed. (There are libraries that make this easy on the PHP side.)

The key feature that allows this to work around firewall issues is the persistent outbound TCP connection from the Arduino to the database. Unlike an HTTP connection which comes and goes for each request, the TCP connection can hang around as long as you like, so the arduino is in effect always listening for another command.

In the BitlashRedisClient example, the data that moves over the wire is Bitlash commands, but of course it can be anything.

Hope this is helpful; good luck with your project.

-br

Rather than a web server and client, use the server and client provided by the ethernet library. It looks as though (have not tried it) it will set up a persistant connection that both ends can write on, so once the arduino has initiated the connection, the Linux box should be able to send to it as long as the connection is kept open.

@ billroy: Sounds like this is what I need. I'll look into it as soon as time permits (somehow my job tends to get in the way) and report back here.
@ wildbill: I'll look into that too. Weekend is nearly here so I might have some time to spend on this.

Thanks,

A

One other thought, do you really need this level of real time response? I assume that this setup is to allow you to turn things on and off remotely. If so, do you really care if it takes a little while to take effect? If not, just have the arduino web client poll the linux box periodically with a get request telling the web server what the current sensor readings are. The web server response will tell the arduino what state it wants the output pins to be in. All user facing display of data is handled on the Linux web site.

I'm afraid time is of the essence here :). Not for all functions (i.e. switching between home- or away-temperatures on a thermostat), but one of the things the Arduino controls is a (3-color-led-) light dimmer. You control a slider in the interface and the dimmer is set accordingly. A slight delay is acceptable, but when it takes a second for the light to react it will take several tries before you reach the right mood/color. And having the Arduino poll the server every 100ms seems like overkill to me...

Agreed, if you need interactive response, polling is a non starter. Sounds like ethernet library all the way - I'll be interested to hear how it goes.

Looks like the weekend is swamped :frowning: But I will post back here if/when I will/will not succeed!

Thanks,

A

Hi,
maybe it's a little redundant now, but this is my idea (worked for me).
It's a quite simple one : you use your arduino as a client, and an altervista site as the server.
In altervista site you create a page in which you can submit a command, the command will be written in another page of the site.
Your arduino will constatly check for new messages in this page (add a message counter, so your arduino will recognize the same order sent multiple times).
If you want, i may "borrow" you my site, it's already ready to be used.
Let me know if you need it :slight_smile:

It's not redundant at all, for me this topic is still alive. I just don't have much time at the moment to resolve it.
Thank you for offering a page to use, but I already have a page on a server (my own VPS). The thing is, I don't want the Arduino polling a server all the time. For a real time response to commands send to the Arduino it would need to poll several times per second. This will cause a lot of traffic 24 hrs/day while it only needs to respond to commands maybe 20 times per day. So I need the Arduino to be a server listening all the time. And to be accessible from outside the LAN. And the ideal situation is getting that without the need for port-forwarding. Hence this topic...

Cheers,

A

Another dont'-touch-the-router approach is to use only UDP packets with known server IP addresses.

See http://ksduino.org/

Sounds interesting, will look in to this as soon as time (read work) allows.

Thanks,

A

OK, I've done some testing. I've set up my linux box to have Apache also listen to a non standard port. Next I wrote a sketch combining client and server sending and receiving on that same port. So when the client on the Arduino connects to my Linux box, it requests a php script that stores the ip address the Arduino is behind and then a second script attempts to connect on that address. I was hoping that since a connection is made on a specific port the "return call" would be forwarded by my router to the Arduino, but no luck. Apparently this ultra simple TCP-hole punching does not fool my router. It works if I forward the port on my router to the Arduino, so my sketch and php script(s) work, but without the port forwarded the router simply ignores the attempts to connect to the Arduino.
I have looked at the examples from ksduino and bitlash but those are a bit beyond my comprehension. I sort of get the Arduino side of those, but the server side is not clear to me.
Next step is trying to do implement UDP hole punching on the Arduino.
Keep you posted,

A

With server code and without port forwarding, you are hitting a brick wall.

You could try moving the Arduino to the public side of your router, but then you would need to use some type of firewall/password code on the Arduino, or every hacker in the world will be turning your lights on and off, or opening and closing you garage door, or whatever the Arduino is supposed to be doing.

What do you mean by 'the public side of the router'?
The security is of course a problem and one of the reasons I'd like to communicate with the arduino without port forwarding. Because if I open a port on my router the arduino is also accessible by hackers. I can use a non-standard port and use 'complicated' POST commands so the arduino only responds to a command with some kind of password build in, but if a hacker wants access he/she will get it in the end...

The public side of the router is the WAN interface (connects to the internet). You could put a switch or hub between your DSL or cable modem and the router, and connect the Arduino to it. You could access it from the localnet of the router, and the internet using the public ip. Some ISPs are kinda fussy about issuing multiple public ips on one modem tho.

Extra difficult if the (cable-)modem and router are combined in one box, as they usually are here on most home-connections.
I'm not giving up on UDP hole punching yet, if only because I learn a lot in the process.

Tnx,

Arjen