I'm a newbie to Arduino but not to programming and controllers in general. I'm trying to build a system that processes some data that is read out of a pump's embedded web processor but it seems to hang after reading about 1500 bytes. The code is fairly straight forward, right out of the examples. See the fragments below.
I can connect and I start to get data (see Output below) but then mangles it a bit and hangs for about 45 seconds (see note in Output) then prints the character count and disconnects. The full output (as pulled from the page on Firefox) is shown below the Output section...
If I change the "server" variable to say, www.google.com, it will pull the whole google page without a problem. The pump's web processor has a number of pages (/net.htm, /cfg.htm, etc.) and the system hangs on each page read at about the same number of characters.
Any thoughts would be greatly appreciated!
Thanks....
Alex...
**** Fragment 1 (in setup) *****
if (client.connect(server, 80)) {
Serial.println("connected");
// Make a HTTP request:
client.println("GET /log.htm HTTP/1.0");
client.println();
}
**** Fragment 2 ******
void loop()
{
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
count++;
}
// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.print("Number of Chars: ");
Serial.println(count);
Serial.println("disconnecting.");
client.stop();
Tried your changes but it still hangs at the same place. I'm wondering if there is some non-ASCII character embedded in the stream that's hanging up the process? Is there a way to dump the stream contents or the buffer (if there is one)?
Maybe you are not reading it fast enough, and the "server" is timing out waiting? Comment out the "Serial.print(c);" statement in the loop and see if it goes past the 1938 count. If it does, and you are using 9600 baud, try increasing the Serial speed.
I guess the main problem is on the server side. Are you able to sniff the traffic between the Arduino and the web server (with a program like WireShark)? It doesn't include a Content-length header, maybe it does some other things wrong.
It's funny because the behavior is the same on all the "pages" within the pump controller. I was wondering about full/half-duplex issues, maybe because the pump and the Ethernet are connected via a gig switch...
I was going to use Wireshark but I need to get a managed switch (or dig up an old hub) so that I can set the ports to be monitored...
Client test code where you can put in the IP address of your pump and see if all the page is returned and displayed in the serial monitor.
//zoomkat 4-04-12
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605
//the arduino lan IP address { 192, 168, 1, 102 } may need
//to be modified modified to work with your router.
#include <SPI.h>
#include <Ethernet.h>
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////
void setup(){
Ethernet.begin(mac, ip);
//Ethernet.begin(mac, ip, gateway, gateway, subnet);
Serial.begin(9600);
Serial.println("Better client test 4/04/12"); // so I can keep track of what is loaded
Serial.println("Send an e in serial monitor to test"); // what to do to test
}
void loop(){
// check for serial input
if (Serial.available() > 0) //if something in serial buffer
{
byte inChar; // sets inChar as a byte
inChar = Serial.read(); //gets byte from buffer
if(inChar == 'e') // checks to see byte is an e
{
sendGET(); // call sendGET function below when byte is an e
}
}
}
//////////////////////////
void sendGET() //client function to send/receive GET request data.
{
if (client.connect(myserver, 80)) { //starts client connection, checks for connection
Serial.println("connected");
client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
client.println(); //end of get request
}
else {
Serial.println("connection failed"); //error message if no client connect
Serial.println();
}
while(client.connected() && !client.available()) delay(1); //waits for data
while (client.connected() || client.available()) { //connected or data available
char c = client.read(); //gets byte from ethernet buffer
Serial.print(c); //prints byte to serial monitor
}
Serial.println();
Serial.println("disconnecting.");
Serial.println("==================");
Serial.println();
client.stop(); //stop client
}
If I had to guess what your problem is, it would be this:
The w5100 IC does not support jumbo frames. The rx buffers are not large enough.
edit: If there is a router between the devices with the MTU set normally (around 1500), it should send standard packets, but with custom equipment, that is not a guarantee.
When a device gets a packet that is too big, the device SHOULD return a "fragmentation required" message back to the sending device. The w5100 apparently can't do that.
OK, so much for apparently. This is in the "Features" section of the Wiznet W5100 datasheet:
Zoomcat: I'll give the code a try. Why the wait for "e" code though?
SurferTim: Interesting thought about the Jumbos. I'll see if I can devise a test to find out. I wonder if I put a "proxy" between the Arduiono and pump to see if that helps. Maybe a simple Firewall can "reframe" the packets...
<a href="cfg.htÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ (seemingly endlessly).
<p>It seems to overrun the buffers or spaz out the serial print function (even at 57600) and I can only regain control by pressing reset on the Ethernet.</p>
<p>ÿ seems to be ASCII 152...</p>
<p>Any thoughts?</p>
<p>Thanks!</p>
<p>Alex...</p>
if (count>=1938){
client.println(); <============Note this...
count=0;
}
/End Fragment************************/
I get more data out of the pump...
If you remember, the system would freeze at 1938 characters which is why I set the "count" test to that. I lose some data from the 1939th character but it picks up at some point and continues to the end of the "page". I can lower the counter test to ridiculously low numbers (like 200) and it still works. If I go above 1938, then the old problem reappears; it locks up...
I've also tried using client.println("User Agent: Mozilla/5.0"); in the setup but it doesn't make a difference... I've also run "Serial" from 9600 to 115200 and it doesn't affect the results...
No, that's 255 or better -1 and means that you're reading an empty buffer.
The problematic part is
while (client.connected() || client.available()) { //connected or data available
char c = client.read(); //gets byte from ethernet buffer
Serial.print(c); //prints byte to serial monitor
}
You're reading from client even if available() returns false, as long as connected() returns true.
Have you tried getting the page without printing every character to the serial interface? Maybe it's a timing issue and the pump is a bit picky about that.
while (client.connected() || client.available()) { //connected or data available
char c = client.read(); //gets byte from ethernet buffer
Serial.print(c); //prints byte to serial monitor
}
is part of Zoomcat's sketch from above; there is a prior while clause that waits for the client.connected() or client.available().
Even with a simple:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
the process fails after 1938 characters read.
How would one read in the entire page without printing it? Load a buffer or array?
Any thoughts on why sending the periodic client.println(); tickles the pump to emit more?
The issue may be with using the serial monitor to view the output. The below code downloads wave period data and extracts the desired data. printing the raw to the serial monitor is commented out and the code seems to work each time it is run. If the raw data is uncommented, sometimes the download is halted, so displaying the raw data in the serial monitor might be part of the issue.
//zoomkat 12-22-10
//simple ethernet client test code
//for use with IDE 0021 and W5100 ethernet shield
//modify the arduino lan ip address as needed
//open serial monitor to see what the arduino receives
//push the shield reset button to run client again
#include <SPI.h>
#include <Ethernet.h>
String readString, readString1;
int x=0;
char lf=10;
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 238, 27 }; // NOAA
EthernetClient client; //(server, 80);
void setup()
{
Ethernet.begin(mac, ip);
Serial.begin(9600);
Serial.println("starting simple arduino client test");
Serial.println();
Serial.println("connecting...");
if (client.connect(server, 80)) {
Serial.println("connected");
client.println("GET /data/5day2/44013_5day.txt HTTP/1.0");
client.println();
} else {
Serial.println("connection failed");
}
}
void loop()
{
if (client.available()) {
char c = client.read();
//Serial.print(c); // uncomment to see raw feed
if (c==lf) x=(x+1);
if (x==14) readString += c;
//readString += c;
}
if (!client.connected()) {
client.stop();
Serial.println("Current data row:" );
Serial.print(readString);
Serial.println();
readString1 = (readString.substring(41,43));
Serial.println();
Serial.print("DPD sec: ");
Serial.println(readString1);
Serial.println("done");
for(;;);
}
}
