Get Arduino data from external web site [SOLVED]

Hello,
I would like to get Arduino data from an external web page, but that doesn’t work :
Here is my test :
IP for my card is 192.168.1.140
My external IP address is hiden in the following code (xx.yy.aabb:zzz). The port zzz is routed on my Arduino card. My IP address is fixed.
On the Arduino card, I’ve a simple server web sketch. It works very well when I access it directly from a browser on my intranet (access to 192.168.1.140) AND on the internet (access to xx.yy.aabb:zzz).

I’ve also a php web page with the current code :

<?php echo" test to Arduino "; echo ""; echo " \n"; $url = "http://xx.yy.aabb:port?aaa"; //**test1** : that works from local php server but nor from external server. //$url = "http://192.168.1.140?aaa"; // **test2** : that works when I try this test from local php server //$url = "http://www.google.fr"; // **test3** : that works from local and external php server if ($GO<>""){ $tab= file($url); foreach ($tab as $str) echo $str; } echo "\n"; echo "\n"; ?>

I’ve installed this php script on my local php server and all works well (test1 to test3).
I’ve also installed this script on site40.net : test 3 works well (I get data so I know the file(URL) function is working), but test 1 is not working.
On the Arduino sketch, there is a Serial.print(c) to the monitor : by this way I can see what I get.
From the site40.net (and test1) I don’t receive anything !

I don’t understand why that don’t work : what difference is there between a request from a browser and a request from the php “file(url)” ?

My external IP address is hiden in the following code (xx.yy.aabb:zzz). The port zzz is routed on my Arduino card. My IP address is fixed.

I would suspect it is the redirect from the public ip to the internal (localnet) address in the router. You have only that one port routed to the Arduino? Do you change the port on the redirect?

Of course I've redirect the zzz port to my Arduino.
I thing the port zzz is correctly redirected to my card because, as explained in my first message, I perform to access the Arduino directly from a web browser with the external address xx.yy.aabb:zzz

lesjno:
Of course I've redirect the zzz port to my Arduino.
I thing the port zzz is correctly redirected to my card because, as explained in my first message, I perform to access the Arduino directly from a web browser with the external address xx.yy.aabb:zzz

Is this web browser in a computer on your localnet? Or is it on another network?

And you are certain that your ISP isn't blocking that port zzz? Mine blocks most ports (ftp,http,smtp,etc) inbound to residential ip addresses.

Effectively when I use my web browser on the local network that works, but when it's not on the local network that don't work. (Of course in both cases I use the external IP address, not the local 192.168... address).

I've tried to change the port : a already use the 8183 for another system, so I tried to use the 8184 but it's the same : that don't work.

Normally the 8xxx ports are not blocked. Are you certain there is nothing in your firewall blocking it? I'm just checking everything. There must be something blocking it somewhere between your public network and your localnet.

I've a freebox and normaly there is no firewall into the box (but I will chek). And there is no firewall on the Arduino ...

I use Mikrotik stuff. I was notorious for locking myself out of my routers with firewall rules. I'm much better at it now. :slight_smile:

If there is a way to get onto your public network locally with a computer, that would be the best test. I have multiple public ips for testing this kind of stuff. That is why I must edit a lot of my code before posting it.

With no arduino code posted and little hardware details, you might try the below simple client code to see if you get a response from the server.

//zoomkat 12-08-11
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605 

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////

void setup(){

  Ethernet.begin(mac, ip);
  //Ethernet.begin(mac, ip, subnet, gateway);
  Serial.begin(9600); 
  Serial.println("Better client test 12/01/11"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(myserver, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

Thanks for your answer, Zoomkat.

When I test your code without changing anything, I get the "connection failed" message.

If I configure gateway and subnet :
byte gateway[] = { 192, 168, 1, 254 }; // internet access via router
byte subnet[] = { 255, 255, 255, 0 }; //subnet mask

and use the " Ethernet.begin(mac, ip, subnet, gateway);" that work :
the result is :
Better client test 12/01/11
Send an e in serial monitor to test
connected
HTTP/1.1 200 OK
Date: Sun, 04 Mar 2012 20:06:55 GMT
Server: Apache
Last-Modified: Sat, 13 Nov 2010 16:31:40 GMT
Accept-Ranges: bytes
Content-Length: 51
Connection: close
Content-Type: text/plain; charset=UTF-8

Woohoo! Your arduino ethernet client works!
zoomkat
disconnecting.

In fact I already used the server and client configuration in my sketch and I know the client works.
But the connection is initiated by the Arduino, so a sort of tunel is mounted between the Arduino and the external server.
I suppose it's a little different when I try to connect directly from the external network : there is no tunel so my box must route to the Arduino...

Hi

It still doesn’t work, but I’ve done other tests and now I’m sure the problem comes from the Arduino side.
In fact I was not sure my box (Freebox) or somthing else on the external network didn’t block ports or not coming from the internet. So I went at a friend home where there is the same box and a system is already connected to the box. This system can be accessed from the internet. The system force a fix IP (192.168.1.210). The router is simply configured to route a port on this address (no association between MAC@ and IP@ within the router).
My sketch is a very basic server test. I print on the monitor all characters I get from the client.
So here was my test :

  • 1 - I Checked I can access the current system of my friend from the internet (outside from the LAN) => result : OK
  • 2 – Command Ipconfig from a PC on the LAN to get all information : gateway, mask, DNS
  • 3 - I configured the Arduino server to use the same information : IP , gateway, mask & DNS.
  • 4 – local test on the LAN IP@ => result OK.
  • 5 – local test from the LAN but with the external IP@:port => result OK
  • 6 – Test from outside the LAN on the same external IP@:port => result NOT OK
  • 7 – I remove the Arduino an I connect the initial system
  • 8 – Same test than the last one (6) => result OK.

Strange isn’t it ?

So there is probably something wrong in the Arduino. But what !

I’ve also test the client configuration and that works : The Arduino can send information to an external php script (outside from the LAN), the site answer (simple echo “OK”) and this answer is received by the Arduino. In this configuration the Arduino receive information from ouside. But in this test the initial connection is initiated but the Arduino. So it's different than the server configuration.

For info, within my Arduino sketch I print the IP, Gateway, Mask and DNS information. (I print the following values : Ethernet.localIP(); Ethernet.ubnetMask(); Ethernet.gatewayIP(); Ethernet.dnsServerIP():wink:

I’ve done a test with DHCP (and just Ethernet.begin(mac)) : Gateway, mask and dns are good.
When I force all parameters, they are used.

I’v also done a test with Ethernet.begin(mac,IP) (so I force the IP@ only) : in this case the gateway and dns are wrong (192.168.1.1) : I don't know whether it's normal or not ... maybe this information can help someone to suggest another idea ?

I don’t see now what I can do !

For info, here is my test sketch :

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
 IPAddress ip(192,168,1, 140);
 //IPAddress ip(192,168,1, 210);
 byte mac[] = { 0x00, 0x16, 0x38, 0x55, 0x07, 0xF6 }; //00:16:38:55:07:f6
 


  IPAddress dns (212,27,40,240);
  IPAddress gateway  (192,168,1,254);  
  IPAddress subnet (255, 255, 255, 0); 


// Initialize the Ethernet server library
// with the IP address and port you want to use 
// (port 80 is default for HTTP):
EthernetServer server(80);

void setup()
{
    Serial.begin(115200);

  int status;
  Serial.println("Connection ...");
  /*
  status = Ethernet.begin(mac);
  if (status != 1){
    Serial.println ("DHCP connection failed.");
    Ethernet.begin(mac,ip,dns,gateway,subnet);
  }
  //*/
  Ethernet.begin(mac,ip,dns,gateway,subnet);
  
  Serial.print("My IP address: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.localIP()[thisByte], DEC);
    Serial.print("."); 
  }
  Serial.println();

  
  // print your local subnetMask:
  Serial.print("My subnetMask: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.subnetMask()[thisByte], DEC);
    Serial.print("."); 
  }
  Serial.println();
  
  // print your local IP gateway:
  Serial.print("My IP gateway: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.gatewayIP()[thisByte], DEC);
    Serial.print("."); 
  }
  Serial.println();
  
  // print your local IP dnsserver:
  Serial.print("My IP dnsServer: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.dnsServerIP()[thisByte], DEC);
    Serial.print("."); 
  }
  Serial.println();



  Serial.println("Connection done");

  server.begin();
}

void loop()
{
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.print(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();

          // output 
          client.println("connection OK");
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } 
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
  }
}

If you do not use dhcp, then the ethernet library does assign all those parameters.

If you specify just the ip, then the library creates a gateway ip by using your ip, and replacing the last digits with a '1'. So if you use 192.168.1.140 for your ip and do not specify the gateway, then it will use 192.168.1.1 for the gateway.

If you do not specify a dns server, it will use the gateway ip of 192.168.1.1 for the dns server.

If you do not specify a subnet mask, it will use 255.255.255.0 for the subnet.

Does this info help?

Thanks for the info !
It doesn't help me to fix my problem, but I understand now why I get such values.
I always use the full method with all the parameters, so normaly all should be good.

I will try to put a sniffer on my network ... maybe I'll get some other interesting info. Because normally the Arduino should receive something from my box ! But in the sketch I can print only data at the upper sofware level. I suppose there a other software level between the "client.read()" and the lower level (where the ethernet signal is received). Or do you know whether it's possible to activate traces into the Arduino ?

So here was my test :

  • 1 - I Checked I can access the current system of my friend from the internet (outside from the LAN) => result : OK
  • 2 – Command Ipconfig from a PC on the LAN to get all information : gateway, mask, DNS
  • 3 - I configured the Arduino server to use the same information : IP , gateway, mask & DNS.
  • 4 – local test on the LAN IP@ => result OK.
  • 5 – local test from the LAN but with the external IP@:port => result OK
    - 6 – Test from outside the LAN on the same external IP@:port => result NOT OK
    - 7 – I remove the Arduino an I connect the initial system
    - 8 – Same test than the last one (6) => result OK.

I'm not sure if I understand all this. If #8 was tested on your friend's router, and that is what you refer to as the "initial system", then you have problems with your router's destination NAT, firewall, or the Arduino network settings.

edit: Maybe I am confused. Is the "initial system" another device with the same ip that is also a server?

Add: This was my problem years ago when I started this kind of stuff. I tested a web server on a specific ip, and all worked well, so I knew that my setup was ok.

Then I replaced that server with another server that had been tested on the localnet. No joy. I could not reach it from the internet, nor could I get internet access.

So I connected the orginal server, and guess what? All was fine. ??

Then I replaced the good server with my bad server, and it failed. :frowning:

Then I powered down my router, waited a few seconds, then plugged it back in. Now the "bad server" is the good one, and the former "good server" will no longer respond to internet requests, nor get internet access.

It ended up being the ARP table in the router. It only did an ARP scan every few hours, so it would not recognize the new ip/mac of the new server for a while. Resetting the router took care of that.

Just a thought...

The "initial system" is a specific card (with ethernet server included) allowing my friend to switch ON & OFF several equipments.
So, at the beginning of my test, I checked I can connect it from outside, to be sure. Then I remove it.
After configuration of the Arduino to set the exact same parameters than the "initial system" (IP, Gateway, etc...) I plug the ethernet cable to the Arduino and test the local and remote connection (points 4, 5 & 6).
At the point 7 : I remove my Arduino card and I plug back the ethernet cable to the "initial system" and check again (at point 8) the remote connection was still working.

Did you see my "Add:" above? Set up the Arduino, then power cycle your router, waiting a few seconds with it off. Try again.

Thanks ! Effectively I didn't see your add. The problem you had seems to be mine !
It's a very good idea ! I will try this when I'm at home ...
I'll keep you informed tonight or tomorrow...

Let me know how it goes.

BTW, I made that troubleshooting seem like I figured it out immediately. That was NOT THE CASE! That fix involved many dozen tries and a lot of cursing! :slight_smile:

I will do this, but I wonder ...
If I change the MAC@ and the IP of the Arduino, that should be the same than switch Off and On the box, no ?

You can lie to the router by using the same mac address on both devices, but this comes at a cost. You can no longer use both devices connected to that router. One will fail.

edit: If you use a different mac and a different ip, there should be no problem. Just route to the new ip. The problem arises when you use the same ip with a different mac address.