Arduino Ethernet Shield as WebClient-proxy server requiring authentication

Hi! I am new to Arduino and as you can see this my first post here.

I am using the following hardware:

  1. Arduino Mega 2560 R3
  2. Ethernet Shield R3

Setup:
I have mounted the Ethernet Shield on top of the Mega, have neither used any jumpers nor bent any pins since I read that it is not necessary to do so the new shield.

I want to use this setup to access some webpages. The problem is that I am using the ethernet connection present in a University which requires a proxy server with authentication. Also, every room in the dorms has it's own static IP, subnet mask, gateway etc.
I am unsure of what extra lines I have to include in my code so as to get past the proxy which requires authentication.

I am presently just trying to run the example WebClient sketch included in the Ethernet Library.

Any help would be appreciated. Please point it out if I have missed some details.

I found a similar question here, but the procedure was not detailed enough for me to understand, nor did anyone confirm if it works.
http://forum.arduino.cc/index.php/topic,13890.0.html

I'm not understanding the network environment you're trying to fit into. You say each room has an IP address. Does this mean that each room has a router that is NATing several IP addresses within the room onto a single IP externally?

Where is the requirement for using a secure proxy server imposed? Typically proxy web servers would only be used to restrict access out of a private network.

I assume the web server would be within your room. Where would your clients be, in networking terms? (Same room? Different rooms? Public internet?)

If you use a computer with your room connection, do you have to change the settings in the computer's browser settings to make an outside connection? Try the below code and see if it works.

//zoomkat 9-22-12
//simple client test
//for use with IDE 1.0.1
//with DNS, DHCP, and Host
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//remove SD card if inserted

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address

char serverName[] = "web.comporium.net"; // zoomkat's test web page server
EthernetClient client;

//////////////////////

void setup(){

  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }

  Serial.begin(9600); 
  Serial.println("Better client test 9/22/12"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(serverName, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println("Host: web.comporium.net");
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

zoomkat:
If you use a computer with your room connection, do you have to change the settings in the computer's browser settings to make an outside connection? Try the below code and see if it works.

//zoomkat 9-22-12

//simple client test
//for use with IDE 1.0.1
//with DNS, DHCP, and Host
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//remove SD card if inserted

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address

char serverName[] = "web.comporium.net"; // zoomkat's test web page server
EthernetClient client;

//////////////////////

void setup(){

if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }

Serial.begin(9600);
  Serial.println("Better client test 9/22/12"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  } 
}

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(serverName, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println("Host: web.comporium.net");
    client.println(); //end of get request
  }
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor
  }

Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

In order to access the internet via a computer in the room, I have to go to Network & Sharing Center --> Change Adapter Settings-->Select Realtek ethernet adapter--> Select IPv4--> Click on properties--> Set my IP address ( provided by the University for every specific room), set Subnet Mask, set Default Gateway, set Primary and Secondary DNS server.

Further, I go to Internet Explorer--> Internet Properties--> Connections-->LAN Settings--> Set up a proxy server with its adress and port.
Now when I access a webpage, I get a pop up windows which asks me to enter a User ID and a Password, after which I am able to acess the internet.

P.S. The code you gave does not work, it says failed to configure using DHCP.
I also tried manually setting the IP, Gateway etc. but then, no connection, probably due to lack of proxy settings

PeterH:
I'm not understanding the network environment you're trying to fit into. You say each room has an IP address. Does this mean that each room has a router that is NATing several IP addresses within the room onto a single IP externally?

I am sorry I do not have sufficient technical knowledge to completely understand what you are asking here. I will however try to get in touch with others here who may know this sort of thing.

PeterH:
Where is the requirement for using a secure proxy server imposed? Typically proxy web servers would only be used to restrict access out of a private network.

Yes, I can access the intranet without proxy server, which is only used to restrict access outside the private network.

PeterH:
I assume the web server would be within your room. Where would your clients be, in networking terms? (Same room? Different rooms? Public internet?)

Again, I will try to find someone who knows the answer to this.

The server: Proxy server of university with authentication.
The client: Arduino with Ethernet Library.

johnwasser came up with a solution that works on proxies without authentication.

johnwasser:
Use the proxy server as the "server" part of the GET request and use the full URL as the URI part of the GET request:

Direct connection:

  // if you get a connection, report back via serial:

if (client.connect("www.actualserver.com", 80)) {
   Serial.println("connected");

// Make a HTTP request through the connected server:
   client.println("GET /search?q=arduino HTTP/1.0");
   client.println();
 }





Proxy connection:


// if you get a connection, report back via serial:
 if (client.connect("www.proxy.com", 8080)) {  // This is connecting to the proxy
   Serial.println("connected");

// Make a HTTP request through proxy:
   client.println("GET http://www.actualserver.com/search?q=arduino HTTP/1.0");
   client.println();
 }

The Ethernet Library does support proxies but not proxies with authentication. Now you need fill in the gap. The Arduino Ethernet Library is not alone here, the iOS device, Iphone/Ipad has same problem ( support proxies but has bug at proxies with authentication). The solution is add an other proxy server with pre-load authentication credentials. Now you should look around see who is able use iOS device at university's network and kneel for share the setting, or setup proxy server for Arduino and share the setting with them. You could make yourself popular.

Check school policy before you start share.

2 common types proxy authentication (a lot of more types, but not for school);-

  • basic authentication, login/password
  • ip authentication, intranet ip address

School might support iOS device via ip authentication, now you need bring an Iphone ( old model, the new model with latest iOS might fix the bug) ask the IP address setting, then transfer it to Arduino with johnwasser's solution.

You should download Wireshark and learn how to use it. You need to be able to figure out what kind of proxy authentication is required; there are so many ways it can be done.

Microsoft has a good example using NTLM that shows the style of the messages going back and forth. While I'm not saying your proxy server is using NTLM (which is popular in the Microsoft world) it's assuredly using some type of encrypted authorization and that's going to prove a big stumbling block for you.

Hi everybody!

After a lot of time, I succeeded.

My proxy server also needs authentication, so I used the logic of this guy:

https://shantamraj.wordpress.com/ (in the middle of the page).

The clues: You need identify the specific server for the ip of arduino (my university used different servers depending of the IP), and the port used by the server.

Also you need to translate the user and password (I used base64 encoded), and (I'm not sure of this) it is better use the IP of the server and save it in a byte format (something like byte serverUniversity[] = {XXX,XX,XX,XX})

hopefully it will be helpful.

Najelandra:
Hi everybody!

After a lot of time, I succeeded.

My proxy server also needs authentication, so I used the logic of this guy:

https://shantamraj.wordpress.com/ (in the middle of the page).

The clues: You need identify the specific server for the ip of arduino (my university used different servers depending of the IP), and the port used by the server.

Also you need to translate the user and password (I used base64 encoded), and (I'm not sure of this) it is better use the IP of the server and save it in a byte format (something like byte serverUniversity[] = {XXX,XX,XX,XX})

hopefully it will be helpful.

Could you please share the code for reference.