Problem with Ethernet client and network Firewall / configuring proxies

Hi everyone,

I am having issues using an arduino uno and ethernet shield (using mac osx 10.8.3) on a university network, and after searching around a bit I am none the wiser.

My problem: I have written an http client that works great on my home network, but fails on the university network. It seems to be getting DHCP service, but after getting the IP address (using DHCP), nothing else happens. One of the system tech's here said it might be the firewall, and recommended configuring a web server to get around it. I dont know too much about networking, so I don't really know what that means, but understand this would some 3rd party software running on a server? Is there any way of getting around this in the Arduino code itself?

On a related note, I was previously using a wifly which I abandoned after what I believe were problems with the university proxy server. In processing you can easily negotiate this with the System.util stuff, but I havent seen anyway of doing this in arduino.

If anyone can point me in the right direction, or offer some advice, it would be much appreciated.

Apologies if I've not given enough info - happy to post with more.

Thanks!

If anyone can point me in the right direction, or offer some advice, it would be much appreciated.

Posting your code that is not working is the usual start.

Fair enough, I didn’t post the code initially because, as mentioned, it works fine on other networks.

Here it is:

/*
 * http client that get new zealand 'felt' earthquake data from http://www.geonet.org.nz/quakes/
 * Parses returned XML to get magnitude and approximate location from XML title
 * Based on Twitter Client example from Arduino.cc
 */
#include <SPI.h>
#include <Ethernet.h>


// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 
  //0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x01 };
  0x90, 0xA2, 0xDA, 0X0D, 0XB1, 0XA7}; //FROM STICKER ON BOTTOM OF SHIELD
IPAddress ip(192,168,1,20);

// initialize the library instance:
EthernetClient client;

const unsigned long requestInterval = 60000;  // delay between requests

char serverName[] = "www.geonet.org.nz";

boolean requested;                   // whether you've made a request since connecting
unsigned long lastAttemptTime = 0;            // last time you connected to the server, in milliseconds

String currentLine = "";            // string to hold the text from server
String xmlData = "";                  // string to hold the tweet
String previousXmlData = "";
boolean readingXmlData = false;       // if you're currently reading the tweet
String magnitudeString;
//String [] dataBits;

int ledPin = 12;

//Motor
int motorPin = 9;
int numbers [10] = {
  0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
int decNumbers [10] = {
  48, 49, 50, 51, 52, 53, 54, 55, 56, 57};

void setup() {
  pinMode (ledPin, OUTPUT);
  // reserve space for the strings:
  currentLine.reserve(256);
  xmlData.reserve(150);

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }


  // attempt a DHCP connection:
  Serial.println("Attempting to get an IP address using DHCP:");
  if (!Ethernet.begin(mac)) {
    // if DHCP fails, start with a hard-coded address:
    Serial.println("failed to get an IP address using DHCP, trying manually");
    Ethernet.begin(mac, ip);
  }
  Serial.print("My address:");
  Serial.println(Ethernet.localIP());
  connectToServer();
}


void loop()
{
  if (client.connected()) {
    //delay (2000);
    if (client.available()) {
      // read incoming bytes:
      char inChar = client.read();

      // add incoming byte to end of line:
      currentLine += inChar; 

      // if you get a newline, clear the line:
      if (inChar == '\n') {
        currentLine = "";
      } 
      //if the current line ends with <item>, it will
      // be followed by the title:
      if ( currentLine.endsWith("<title>Magnitude ")) {
        // tweet is beginning. Clear the tweet string:
        readingXmlData = true; 
        xmlData = "";
      }
      // if you're currently reading the bytes of data,
      // add them to the data String:
      if (readingXmlData) {
        if (inChar != '<') {
          xmlData += inChar;
        } 
        else {
          // if you got a "<" character,
          // you've reached the end of the tweet:
          readingXmlData = false;
          Serial.print("xmlData:");
          Serial.println(xmlData); 
          // close the connection to the server:
          client.stop(); 
          //check its different
          if (xmlData != previousXmlData){//
          Serial.println("NEW EARTHQUAKE");
            // CHECK FOR PHRASE 'CHRISTCHURCH'
            if(xmlData.endsWith("Christchurch")){
              // does it say 'christchurch'?
              parseData(xmlData);
              previousXmlData = xmlData;
              Serial.print("previousXmlData:");
              Serial.println(previousXmlData); 
            }
            else{
              Serial.println("EARTHQUAKE NOT LOCATED NEAR CHRISTCHURCH");
            }
          }else{
            Serial.println("NO NEW EARTHQUAKES");
          }
        }
      }
    } 
   else if (!client.available()){
    //tweet status CLIENT NOT AVALABLE
    Serial.println ("client not available");
    delay (5000);
   } 
  }
  else if (millis() - lastAttemptTime > requestInterval) {
    // if you're not connected, and two minutes have passed since
    // your last connection, then attempt to connect again:
    connectToServer();
  }
}

void connectToServer() {
  // attempt to connect, and wait a millisecond:
  Serial.println("connecting to server...");
  if (client.connect(serverName, 80)) {
    Serial.println("making HTTP request...");
    // make HTTP GET request:
    client.println("GET /quakes/services/felt.rss HTTP/1.1");
    client.println("HOST: www.geonet.org.nz");
    client.println();
    Serial.println ("Just Connected");
  }
  // note the time of this connect attempt:
  lastAttemptTime = millis();
}
// EXAMPLE RESPONSE:  2.3, Wednesday, March 6 2013 at 12:06:18 am (NZDT), 5 km east of Porangahau
void parseData(String data){
  // parse the earthquake title to get data
  //String dataArray [] = split(data, ',');
  String buffer = "";
  //buffer = data.toCharArray(buffer, data.length());
  Serial.print("data length: ");
  Serial.println(data.length());
  for (int i=0; i<data.length();i++){
    char x = data.charAt(i);
    if (x != ',') {
      buffer += x;
    } 
    else {
      // if you got a "," character,
      // you've reached the end this bit of data:

      if(buffer.length() == 4){
        magnitudeString = buffer;
        magnitudeString.trim();

        byte magBytes[magnitudeString.length() + 1];
        magnitudeString.getBytes(magBytes, magnitudeString.length() + 1);
        int magIntBuffer;

        char magCharBuff [4]; 
        magnitudeString.toCharArray(magCharBuff, 4);
        float magFloat = atof(magCharBuff);
        Serial.print ("magFloat: ");
        Serial.println (magFloat);
        int magInt = map(magFloat*100, 100, 800, 100, 255);
        //
        int durationInt = map(magInt, 100, 255, 1000, 5000);
        Serial.print ("duration: ");
        Serial.println (durationInt);
        driveMotor(magInt, durationInt);
      }
      else{
        // ignore
      }
    }
  }
}

So is it possible that there’s an error that gets through other networks, but causes it to fail on this particular one? Might it be caused by specifying an IP address and then attempting to get one with DHCP?

Other details - its the Official Arduino ethernet shield with POE

I have written an http client that works great on my home network, but fails on the university network. It seems to be getting DHCP service, but after getting the IP address (using DHCP), nothing else happens.

Set up Bypass proxy server for local address, or No Proxy for local address.

Firefox

IE/Chrome

if your university go fancy for proxy setting, - Use automatic configuration script, then disable proxy when you are working with arduino and switch back when you need to back on line.

johnwasser came up with a solution that works on proxies. You might want to give it a try. http://arduino.cc/forum/index.php/topic,158690.msg1188318.html#msg1188318

SurferTim: johnwasser came up with a solution that works on proxies. You might want to give it a try. http://arduino.cc/forum/index.php/topic,158690.msg1188318.html#msg1188318

good catch, this solution works for Arduino http client which OP try to do, mine is for Arduino http server.

Thanks very much for your replies everyone - I'll try your suggestions out and let you know how it goes!

Simple client test code that you can try to see if it will connect. Does the university instruct the students to set their browsers for proxies?

//zoomkat 2-13-12
//DNS and DHCP-based web client test code
//for use with IDE 1.0
//open serial monitor and send an e to test
//and to see test result
//for use with W5100 based ethernet shields
//browser equivelant URL: 
// http://web.comporium.net/~shb/arduino.txt
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605 

#include <SPI.h>
#include <Ethernet.h>
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
char serverName[] = "web.comporium.net"; // zoomkat's test web page server
EthernetClient client;

//////////////////////

void setup(){
  Serial.begin(9600); 
  Serial.println("DNS and DHCP-based web client test 2/13/12"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }
  // print your local IP address:
  Serial.print("Arduino IP address: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.localIP()[thisByte], DEC);
    Serial.print("."); 
  }
  Serial.println();
  Serial.println();
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(serverName, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

Hi again - thanks for your suggestions its been a great help, and my code is now working on the university network.

Zoomkat, your test worked well, but it would only work with the mac address you provided, not the one i had been using for this arduino. Now that I changed my setup() code to match yours it still works well, but again only with that mac address. I don’t really understand why as this mac address has worked previously.

Is it the following section that makes the difference?

Serial.print("Arduino IP address: ");
  for (byte thisByte = 0; thisByte < 4; thisByte++) {
    // print the value of each byte of the IP address:
    Serial.print(Ethernet.localIP()[thisByte], DEC);
    Serial.print(".");