Strange behavior and problem with client.read in Ethernet Library...

Hi all,

I’m a newbie to Arduino but not to programming and controllers in general. I’m trying to build a system that processes some data that is read out of a pump’s embedded web processor but it seems to hang after reading about 1500 bytes. The code is fairly straight forward, right out of the examples. See the fragments below.

I can connect and I start to get data (see Output below) but then mangles it a bit and hangs for about 45 seconds (see note in Output) then prints the character count and disconnects. The full output (as pulled from the page on Firefox) is shown below the Output section…

If I change the “server” variable to say, www.google.com, it will pull the whole google page without a problem. The pump’s web processor has a number of pages (/net.htm, /cfg.htm, etc.) and the system hangs on each page read at about the same number of characters.

Any thoughts would be greatly appreciated!

Thanks…

Alex…

**** Fragment 1 (in setup) *****
if (client.connect(server, 80)) {
Serial.println(“connected”);
// Make a HTTP request:
client.println(“GET /log.htm HTTP/1.0”);
client.println();
}


**** Fragment 2 ******

void loop()
{
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
count++;
}
// if the server’s disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.print("Number of Chars: ");
Serial.println(count);
Serial.println(“disconnecting.”);
client.stop();

// do nothing forevermore:
for(;:wink:
;
}
}


Output from "Serial.print()****
connecting…
connected
HTTP/1.0 200 OK
Date: Wed, 05 Apr 2006 02:53:05 GMT
Expires: Wed, 05 Apr 2006 02:52:05 GMT
Cache-control: private
Content-type: text/html

[Title removed for privacy] body { font-family: Arial,Helvetica,sans-serif; color: #5D5E67 } a:link, a:visited { font-weight: normal; color: #5D5E67 } .gh { font-weight: bold; color: #00942F; font-size: 16px; padding-bottom: 12px; } .gh2 { font-weight: bold; color: #00942F; font-size: 14px } td {font-size: 14px; color: #5D5E67;} #nav { background: url('n.gif') repeat-x; width: 970px; height: 58px; margin-top: 74px; } #n2 { padding-top: 33px; padding-left: 385px; } #n2 td { background: url('t.gif') no-repeat; width: 90px; height: 25px; margin-right: 10px; text-align: center; } #n2 td a { font-size: 12px; margin-right: 10px; } #logo { position: absolute; top: 0px; background: url('l.gif') no-repeat; width: 552; height: 87px;} .a{padding-right: 20px;}
Overview Pump Network Log
Overview body { font-family: Arial,Helvetica,sans-serif; color: #5D5E67 } a:link, a:visited { font-weight: normal; color: #5D5E67 } .gh { font-weight: bold; color: #00942F; font-size: 16px; padding-bottom: 12px; } .gh2 { font-weight: bold; color: #00942F; font-size: 14px } td {font-size: 14px; color: #5D5E67;} #nav { background: url('n.gif') repeat-x; width: 970px; height: 58px; margin-top: 74px; } #n2 { padding-top: 33px; padding-left: 385px; } #n2 td { background: url('t.gif') no-repeat; width: 90px; height: 25px; margin-right: 10px; text-align: center; } #n2 td a { font-size: 12px; margin-right: 10px; } #logo { position: absolute; top: 0px; background: url('l.gif') no-repeat; width: 552; height: 87px;} .a{padding-right: 20px;}

Log:

For operation statistics, to go Overview page.

Current error: 0(See error description)

Last error: 1(See error description)

Startup diagnosis: 0000000000000000

Content ©copyright 2011 [Data removed for privacy reasons]

See if this does better.

void loop()
{
  while(client.connected()) {
    while(client.available()) {
      char c = client.read();
      Serial.print(c);
      count++;
    }
  }
  Serial.print("Number of Chars: ");
  Serial.println(count);
  Serial.println("disconnecting.");
  client.stop();

  // do nothing forevermore:
  while(1);
}

SurferTim,

Tried your changes but it still hangs at the same place. I'm wondering if there is some non-ASCII character embedded in the stream that's hanging up the process? Is there a way to dump the stream contents or the buffer (if there is one)?

Thanks again,

Alex.

Maybe you are not reading it fast enough, and the "server" is timing out waiting? Comment out the "Serial.print(c);" statement in the loop and see if it goes past the 1938 count. If it does, and you are using 9600 baud, try increasing the Serial speed.

No luck. Same 1938 characters without the serial out. I tried running at 19200 in my earlier debugging but it made no difference...

I guess the main problem is on the server side. Are you able to sniff the traffic between the Arduino and the web server (with a program like WireShark)? It doesn't include a Content-length header, maybe it does some other things wrong.

It's funny because the behavior is the same on all the "pages" within the pump controller. I was wondering about full/half-duplex issues, maybe because the pump and the Ethernet are connected via a gig switch...

I was going to use Wireshark but I need to get a managed switch (or dig up an old hub) so that I can set the ports to be monitored...

Thanks...

Client test code where you can put in the IP address of your pump and see if all the page is returned and displayed in the serial monitor.

//zoomkat 4-04-12
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605 
//the arduino lan IP address { 192, 168, 1, 102 } may need 
//to be modified modified to work with your router.

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////

void setup(){

  Ethernet.begin(mac, ip);
  //Ethernet.begin(mac, ip, gateway, gateway, subnet);
  Serial.begin(9600); 
  Serial.println("Better client test 4/04/12"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(myserver, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

If I had to guess what your problem is, it would be this: http://en.wikipedia.org/wiki/Jumbo_frame The w5100 IC does not support jumbo frames. The rx buffers are not large enough.

edit: If there is a router between the devices with the MTU set normally (around 1500), it should send standard packets, but with custom equipment, that is not a guarantee.

When a device gets a packet that is too big, the device SHOULD return a "fragmentation required" message back to the sending device. The w5100 apparently can't do that.

OK, so much for apparently. This is in the "Features" section of the Wiznet W5100 datasheet:

Not support IP Fragmentation

Zoomcat: I'll give the code a try. Why the wait for "e" code though? SurferTim: Interesting thought about the Jumbos. I'll see if I can devise a test to find out. I wonder if I put a "proxy" between the Arduiono and pump to see if that helps. Maybe a simple Firewall can "reframe" the packets...

Thanks again!

Zoomcat: I'll give the code a try. Why the wait for "e" code though?

The sending of the e provides am method to control when the get request is sent, and allows for repeated get request without restarting the arduino.

So, the results are in:

(Top cut off as it’s the same as the previous posts)

Overview <a href="cfg.htÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ (seemingly endlessly). <p>It seems to overrun the buffers or spaz out the serial print function (even at 57600) and I can only regain control by pressing reset on the Ethernet.</p> <p>ÿ seems to be ASCII 152…</p> <p>Any thoughts?</p> <p>Thanks!</p> <p>Alex…</p>

A clue?

I spent the last couple of hours playing around and found something interesting. If I insert the following code in the read loop:

/Fragment*******/
void loop()
{
while(client.connected()) {
while(client.available()) {
char c = client.read();
Serial.print(c);
count++;
ccount++;

if (count>=1938){
client.println(); <============Note this…
count=0;
}

/End Fragment************************/

I get more data out of the pump…

If you remember, the system would freeze at 1938 characters which is why I set the “count” test to that. I lose some data from the 1939th character but it picks up at some point and continues to the end of the “page”. I can lower the counter test to ridiculously low numbers (like 200) and it still works. If I go above 1938, then the old problem reappears; it locks up…

I’ve also tried using client.println(“User Agent: Mozilla/5.0”); in the setup but it doesn’t make a difference… I’ve also run “Serial” from 9600 to 115200 and it doesn’t affect the results…

The thick plottens!

Alex…

ÿ seems to be ASCII 152...

No, that's 255 or better -1 and means that you're reading an empty buffer.

The problematic part is

  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

You're reading from client even if available() returns false, as long as connected() returns true.

Have you tried getting the page without printing every character to the serial interface? Maybe it's a timing issue and the pump is a bit picky about that.

The

while (client.connected() || client.available()) { //connected or data available char c = client.read(); //gets byte from ethernet buffer Serial.print(c); //prints byte to serial monitor }

is part of Zoomcat's sketch from above; there is a prior while clause that waits for the client.connected() or client.available().

Even with a simple:

if (client.available()) { char c = client.read(); Serial.print(c); }

the process fails after 1938 characters read.

How would one read in the entire page without printing it? Load a buffer or array?

Any thoughts on why sending the periodic client.println(); tickles the pump to emit more?

Thanks again for all the help!

Alex...

The issue may be with using the serial monitor to view the output. The below code downloads wave period data and extracts the desired data. printing the raw to the serial monitor is commented out and the code seems to work each time it is run. If the raw data is uncommented, sometimes the download is halted, so displaying the raw data in the serial monitor might be part of the issue.

//zoomkat 12-22-10
//simple ethernet client test code
//for use with IDE 0021 and W5100 ethernet shield
//modify the arduino lan ip address as needed
//open serial monitor to see what the arduino receives
//push the shield reset button to run client again

#include <SPI.h>
#include <Ethernet.h>
String readString, readString1;
int x=0;
char lf=10;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 238, 27 }; // NOAA

EthernetClient client; //(server, 80);

void setup()
{
  Ethernet.begin(mac, ip);
  Serial.begin(9600);
  Serial.println("starting simple arduino client test");
  Serial.println();
  Serial.println("connecting...");

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /data/5day2/44013_5day.txt HTTP/1.0");
    client.println();
  } else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    //Serial.print(c);  // uncomment to see raw feed
    if (c==lf) x=(x+1);
    if (x==14) readString += c;
    //readString += c;
  }

  if (!client.connected()) {
     client.stop();

    Serial.println("Current data row:" );
    Serial.print(readString);
    Serial.println();
    readString1 = (readString.substring(41,43));
    Serial.println();
    Serial.print("DPD sec: ");
    Serial.println(readString1);
    Serial.println("done");

    for(;;);

    }
 }