WiFi shield TCP to webserver is very slow

Hi all,

I'm testing a WiFi shield against a web server, and see the following using a test sketch:

Connected to the server in 5ms
Send push in 8ms
Server response OK in 1909/98ms

The 1909ms is the time it takes between the last print to client and data being available from it. 98ms is what it takes for all available data to be read. This is using a web server on the same local network, but roughly the same 2-second delay is observed against a housed server. I have confirmed with tcpdump that it really takes this time between the syn/syn-ack/ack handshake and the push from the shield being received on the server.

Has anyone else observed this, and know how to cure it? Two seconds per request is an eternity for certain applications.

A lot of slow connections are due to excessive packet creation. Each time you use one of the following...

client.print()
client.println()
client.write()

...you create another packet.

My goal is to send one packet if possible. I usually do get away with 4 or less. How are you doing?

Please posted the test sketch if you need more assistance.

Thanks - this is the relevant part of the sketch:

  time = millis();
  // Test connectivity with the server
  while (!serverOK) {
    if (client.connect(server, 80)) {
      delta = millis() - time;
      Serial.print("Connected to the server in ");
      Serial.print(delta);
      Serial.println("ms");
      time = millis();
      client.print("GET /tag.php?id=01020304050607&blank=1&nasid=00-2C-DE-AD-BE-EF");
      client.println(" HTTP/1.1");
      client.println("Host: blah.com");
      client.println("User-Agent: TestClient");
      client.println("Connection: close");
      client.println();
      delta = millis() - time;
      Serial.print("Sent push in ");
      Serial.print(delta);
      Serial.println("ms");
      serverOK = true;
    }
    else {
      Serial.println("Connection to server failed");
      delay(100);
    }
  }
  // Wait for response
  time = millis();  
  while (!client.available()) {}
  time2 = millis();  
  while (client.available()) {
    inChar = client.read();
    inLine += inChar;
    if (inChar == '\n') {
      inLine = "";
    }
  }
  if (inLine == "OK") {
    delta = millis() - time;
    delta2 = millis() - time2;
    Serial.print("Server response OK in ");
    Serial.print(delta);
    Serial.print("/");
    Serial.print(delta2);
    Serial.println("ms");
    client.stop();
  }
  else {
    client.stop();
    Serial.print("Server response:");
    Serial.println(inLine);
    while (true); // Stop here
  }

It's all sent in one packet, and the server replies one packet, all within the MTU.

Additional info - for comparison, I dropped in an Ethernet shield in place of the WiFi shield, and here is the result running the same sketch:

Connected to the server in 2ms
Sent push in 5ms
Server response OK in 39/36ms

Obviously the WiFi shield is having issues... what they are, I don't know.

SurferTim:
A lot of slow connections are due to excessive packet creation. Each time you use one of the following...

client.print()
client.println()
client.write()

...you create another packet.

This is true for the Ethernet shield but we don't know for the WiFi shield. We know the code of the library but that includes just sending the data to the embedded ATmega168 on the shield. Because currently (to my knowledge) no source code for the firmware of the WiFi shield is published, I cannot tell you what the shield is doing with the sent data. My guess is, that it accumulates the data till a packet is ready but I don't know it. The Arduino people promised to open that source code to allow us not only to study it but also to modify it, but that didn't happen yet (or it was quite a very hidden action).

@pylon: And you are probably correct, but 2 seconds is too long. That is an eternity.

@mother: Just out of curiosity, does all the data you sent get through? It just gets there in one big clump, is that it? Can you see how many packets it is sending every 2 seconds? A few or one big one?

I don't use the wifi shield. I use the ethernet shield connected to a Mikrotik router with a wireless card. Very speedy!

Any luck with this? I unfortunately didn't see this problem before purchasing over the New Year. Terrible performance. I gave up - but would love to give it another try if they've fixed the firmware.

Thanks!
p

Has anybody put a server on a pc and run something like wireshark on the pc to study the wifi shield traffic?

Can it be encryption at work?

I dont think so, because i get the same response times when i use an unencrypted Wlan.
Do users with Wifi shields from other brands experience the same issues?
This problem is really annoying, it makes the wifi shield useless…

Deniz

A lot of slow connections are due to excessive packet creation. Each time you use one of the following...

client.print()
client.println()
client.write()

...you create another packet.

so how does one send a lot of data in 1 packet? - i'm still trying to figure this out

Since there is no client.send() command, I presume that each of these creates a separate packet, and is sent as the wifi card gets it, as does the ethernet shield. For example, this will send 4 packets.

      client.println("Host: blah.com");
      client.println("User-Agent: TestClient");
      client.println("Connection: close");
      client.println();

This is the same stuff sent as one packet.

      client.println("Host: blah.com\r\nUser-Agent: TestClient\r\nConnection: close\r\n");

If someone could use Wireshark and confirm this, that would take the presumption out of the equation.

edit: Some user suggested a client.send() command be added to the ethernet library. That would not be a bad idea. That way you could use the device buffer instead of the Arduino SRAM. It would go something like this to send one packet:

      client.println("Host: blah.com");
      client.println("User-Agent: TestClient");
      client.println("Connection: close");
      client.println();
      client.send();

edit2: To maintain legacy compatibility, it would require two new functions. One to prevent the immediate send, and another to send.

       client.nosend();
      client.println("Host: blah.com");
      client.println("User-Agent: TestClient");
      client.println("Connection: close");
      client.println();
      client.send();

I traced the client.print() as far as I can go and each one of them will trigger a call to send SPI command to the shield MCU with the string so I think SurferTim is right. Each print or write creates its separate packet.

BTW, has anyone noticed the limitation of string length being around 90 bytes? If you do print() with a string say 100 bytes long, your other end just doesn't receive anything.

If you look into the firmware source code you'll find the SPI data transfer buffer size to be 100 bytes (ard_spi.c:75). Because this buffer has to hold not only the payload data but also some additional information (about 9 bytes) you'll end with about max. 91 bytes of payload.

#define  _BUFFERSIZE 100

Interestingly there is no check for that limit in the library on the Arduino side and in the protocol the message length is encoded in two bytes which would lead to the expectation that bigger buffer sizes are intended.

pylon,

You are such a great programmer that I am not!!! Well, We now know the original cause of the 90 byte limitation, is there anything that can be done about it? Say increase that number without bringing down the whole firmware? I don't know even how to compile the firmware myself. By the way, does the firmware check the string length when receiving or does it go on the fritz after it unknowingly overruns its buffer? Also which version of firmware did you look at? I know Arduino team is not great at providing software version numbers but file dates are all I wanted to have. FYI, my limit was 90. 91 will not result any data sent.

Karma for you pylon!

Well, We now know the original cause of the 90 byte limitation, is there anything that can be done about it? Say increase that number without bringing down the whole firmware?

Unfortunately not, you have to at least recompile the firmware to increase the value.

By the way, does the firmware check the string length when receiving or does it go on the fritz after it unknowingly overruns its buffer?

Yes, it does check for the buffer size and ignores the whole message if the buffer overruns.

Also which version of firmware did you look at?

I got the latest firmware from github, the Arduino library from IDE 1.0.4.

FYI, my limit was 90. 91 will not result any data sent.

The header and trailer is 9 bytes but if the message completely fills the buffer it will be ignored, so you're right, with 91 it starts to fail.

pylon,

Geez, they should hire you to consult on this while wifi project! I am really not even close to the Arduino team/core to make any contact with them to say, "hey, this code is not working because A-B-C". I'm glad the firmware won't crash on message too long (that'd be lame).

Today (or last night) I found out one more potential problem. If the server crashes (for whatever reason) resetting with reset button may not bring it back. I'm still pinpointing it but with somewhat improved arduino coding, I have to try harder to crash the server side :wink: See, I'm improving a bit too.

We are also experiencing a 1-2 second delay when making basic web connections (client request) with the WIFI shield. Using version 1.0.3 of the Arduino IDE because when we upgraded the firmware of the WIFI shield to use IDE 1.0.5+ it wouldn’t work at all, so we downgraded. The server is returning the response almost instantly and the data being sent is just 48 bytes. Where is the 1-2 second delay in the WIFI shield firmware/software…how can we remove it. We basically want to be doing our normal loop() code until some data is available back from the web request – we can’t have the arduino stalling entirely for several seconds waiting for a web request to return.

Any help would be much appreciated.

Where is your code?