WiFi shield (official) sending, but slowly

I have an official wifi shield, currently on an Uno R2. I have some code that receives a request, does some simple parsing, and returns a simple response. It functioned slower than I expected, so I set up some timers to see what was slow. Turns out that the arduino receives an http message in about 60ms, but sending a similarly sized message back out takes about 4 seconds!

For simplicity, I'll show the code from the WiFiWebServer example, which exhibits the same behavior (the code in my project is very similar):

void loop() {
  unsigned long time = micros();
  //Serial.println("In Loop");
  // listen for incoming clients
  WiFiClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println("Connection: close");
          client.println();
          client.println("<!DOCTYPE HTML>");
          client.println("<html>");
          // add a meta refresh tag, so the browser pulls again every 5 seconds:
          client.println("<meta http-equiv=\"refresh\" content=\"5\">");
          // output the value of each analog input pin
          for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
            int sensorReading = analogRead(analogChannel);
            client.print("analog input ");
            client.print(analogChannel);
            client.print(" is ");
            client.print(sensorReading);
            client.println("
");       
          }
          client.println("</html>");
           break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } 
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
      // close the connection:
      client.stop();
      Serial.println("client disonnected");
      Serial.println("Time: " + String(micros() - time));
  }
}

You can see my timer code added to the example, and the time to send a message over wifi is consistently on the order of 3.7-3.8 seconds.

Ideally I'd like to be able to send my data in ~100ms (remember I receive a similar size in ~60ms, and I think wifi is symmetrical?).

I read here Arduino Forum that each println command is though to create a new packet (unconfirmed, but that's how the ethernet shield works, which shares a lot of the library). I read elsewhere (sorry, didn't save the link) that each character is sent as a separate packet with print or println, and that write should be used instead. If that were the case, I'd expect to be able to drop about 16 tcp bytes for every byte I'm currently sending (minus one), which would speed me from ~4 seconds to ~250ms, much closer to the speed I'm looking for.

So with that in mind, I modified the WiFiWebServer example with the following:

void loop() {
  unsigned long time = micros();
  //Serial.println("In Loop");
  // listen for incoming clients
  WiFiClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);

        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nConnection: close\r\n\r\n<!DOCTYPE HTML>\r\n<html>\r\n<meta http-equiv=\"refresh\" content=\"5\"><body>Hello just 
testing</body></html>");
           break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } 
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
      // close the connection:
      client.stop();
      Serial.println("client disonnected");
      Serial.println("Time: " + String(micros() - time));
  }
}

So now there is only a single client.println statement, which should mean just one packet sent, and a lot of overhead cleared. This reduced the times down to about 1.3s. Big improvement!!

Based on this, I'm thinking that each println/print/write statement does create a new data packet, and all the overhead that goes with that. My messages will be short (~100 characters), so the more overhead I can reduce, the better.

While reducing the number of packets created is good (285% speedup!!), 1.3 seconds is still a lot more than the ~200ms I expected from reducing the number of packets. And it's much greater than the ~60ms times I get receiving data. Are there any known issues limiting the speed of the wifi shield? What else can I do to get a speedup? Most of the information I can find refers to the ethernet shield, and while that was helpful, I'm still looking for more improvement.

One update, I realized that in the two examples I posted, the first one has to go through a loop of analog reads. I removed that code, and got ~1.3 second times even with 9 client.println commands still in place. So the time improvement was all in that loop, NOT in the client.println commands.

Could it be that they are sending each character as a packet? If so I'm still hopeful for my ~200ms times potentially!

For reference, removed the loop from the first example and replaced it with:

client.println("<body>Hello just 
testing</body>");

Found the other page of info: http://arduino.cc/forum/index.php/topic,134868.0.html on the ethernet shield issue.

That does not appear to be my case, since I am only making ~10 calls to client.println, not one per character. Perhaps a little speedup could come from this, but not the 16x speedup I thought. I guess it's back to square one, which is just a question:

How can I make the arduino + wifi shield send data faster?

That does not appear to be my case, since I am only making ~10 calls to client.println, not one per character.

The problem is that client.println() itself does call client.write() for every character in the string you provided. And as far as I remember the shield sends out one packet for every call of the write() method. So try to send as much as possible with one call of client.write(buffer, length), it will speed up things tremendously.

Thank you for pointing this out! I tried it in the sample, and saw little or no improvement, but I decided to try it in my own code anyway as a test. I haven't fully implemented yet, I still have one call to client.println that sends about 30 chars, but I put the other 140 chars or so that were being sent into 3 client.write statements. Time decreased from 3.7 seconds to about 2.2 seconds! I can optimize a little more to send fewer packets, but it will take a slightly deeper rewrite to my code (will be worth it though).
If I take out the client.write completely (not useful, except as a test) the time drops to just over 1 second (~1.02)! That's with 3 client.write statements. That's a good improvement!

Also, while playing around, I put some more timing statements in. You'll see in the sample code, the timer stops after client.stop runs. That command takes about a second on its own. If you don't include that, I'm now down to just a few millisecons for making 3 client.write calls!

Lots of optimization to do, so that my code still does what I need it too, but this is a big improvement. Quantitatively, the page loads seem much faster from a wifi device too. Seems like normal web page loading times, not slow.

The below standard web server setup runs pretty fast when the fast data is selected. Perhaps your delay is just an artifact of the wifi connection with the router.

// zoomkat's meta refresh data frame test page 4/12/13
// use http://192.168.1.102:84 in your brouser for main page
// http://192.168.1.102:84/data static data page
// http://192.168.1.102:84/datastart meta refresh data page
// for use with W5100 based ethernet shields
// set the refresh rate to 0 for fastest update
// use STOP for single data updates

#include <SPI.h>
#include <Ethernet.h>

const int analogInPin0 = A0;
const int analogInPin1 = A1;
const int analogInPin2 = A2;
const int analogInPin3 = A3;
const int analogInPin4 = A4;
const int analogInPin5 = A5;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // arduino ip in lan
byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
unsigned long int x=0; //set refresh counter to 0
String readString; 

//////////////////////

void setup(){
  Serial.begin(9600);
    // disable SD SPI if memory card in the uSD slot
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  server.begin();
  Serial.println("meta refresh data frame test"); // so I can keep track of what is loaded
}

void loop(){
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
         if (readString.length() < 100) {
          readString += c; 
         } 
        //check if HTTP request has ended
        if (c == '\n') {

          //check get atring received
          Serial.println(readString);

          //output HTML data header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();

          //generate data page
          if(readString.indexOf("data") >0) {  //checks for "data" page
            x=x+1; //page upload counter
            client.print("<HTML><HEAD>");
            //meta-refresh page every 1 seconds if "datastart" page
            if(readString.indexOf("datastart") >0) client.print("<meta http-equiv='refresh' content='1'>"); 
            if(readString.indexOf("datafast") >0) client.print("<meta http-equiv='refresh' content='0'>"); 
            client.print("<title>Zoomkat's meta-refresh test</title></head><BODY>
");
            client.print("page refresh number ");
            client.print(x); //current refresh count
            client.print("

");
              //output the value of each analog input pin
            client.print("analog input0 ");
            client.print(" is ");
            client.print(analogRead(analogInPin0));
            client.println("
");
            
            client.print("analog input1 ");
            client.print(" is ");
            client.print(analogRead(analogInPin1));
            client.println("
");
                        
            client.print("analog input2 ");
            client.print(" is ");
            client.print(analogRead(analogInPin2));
            client.println("
");
            
            client.print("analog input3 ");
            client.print(" is ");
            client.print(analogRead(analogInPin3));
            client.println("
");
                                    
            client.print("analog input4 ");
            client.print(" is ");
            client.print(analogRead(analogInPin4));
            client.println("
");
            
            client.print("analog input5 ");
            client.print(" is ");
            client.print(analogRead(analogInPin5));
            client.println("
");
            client.print("</BODY></HTML>");
           }
          //generate main page with iframe
          else
          {
            client.print("<HTML><HEAD><TITLE>Zoomkat's frame refresh test</TITLE></HEAD>");
            client.print("Zoomkat's Arduino frame meta refresh test 4/12/13");
            client.print("

Arduino analog input data frame:
");
            client.print("&nbsp;&nbsp;<a href='http://192.168.1.102:84/datastart' target='DataBox' title=''yy''>META-REFRESH</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/data' target='DataBox' title=''xx''>SINGLE-STOP</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/datafast' target='DataBox' title=''zz''>FAST-DATA</a>
");
            client.print("<iframe src='http://192.168.1.102:84/data' width='350' height='250' name='DataBox'>");
            client.print("</iframe>
</HTML>");
          }
          delay(1);
          //stopping client
          client.stop();
          //clearing string for next read
          readString="";
        }
      }
    }
  }
}

rmurrish,

Thanks for the testing etc. I wanted to know what revision your WiFi shield is. Mine is R3. I can't even send a string more than 90 bytes even if I wanted it. The other end just receives nothing. A couple things to clarify:

I read here Arduino Forum that each println command is though to create a new packet (unconfirmed, but that's how the ethernet shield works, which shares a lot of the library).

That is correct. I traced the code to a function that sends SPI command followed by the string, and its length. That other side is inside the firmware so I don't know about it but am pretty sure each print and each write creates a new packet.

The problem is that client.println() itself does call client.write() for every character in the string you provided. And as far as I remember the shield sends out one packet for every call of the write() method. So try to send as much as possible with one call of client.write(buffer, length), it will speed up things tremendously.

Only the write(buffer,length) was correct. There is no problem with client.println() and it DOES NOT call client.write(). As a matter of fact, it is the other way. A write(char) calls write(c-string,lengh=1), a println calls write(c-string, strlen(c-string).

Take a look at the following sequence diagram I created, trying to solve the same speed problem:
First line is the "normal" sequence that pylon was expecting for everything else that derives from Print.
Next two lines are for WiFi methods.

liudr is right about the char* prints and the OP is using them. What I had in mind was the call with the F() macro (__FlashStringHelper) which is often used to save RAM. This one still calls write(char) for every character in the constant string and is wasting lot's of network resources.

pylon,

I didn't consider the flash helper. I will add it into my diagram when I have time. Good point!

Thank you for sharing what you know. My WiFi Shield is an R3.

I don't know anything about the flash helper, so if that's relevant to me, some explanation would be great!

Flash helper:

F("")

Eg. Serial.print("hello"); // not using flash helper
Serial.print(F("hello")); // using flash helper

This will save hello in flash so you can preserve SRAM. But according to pylon, this function uses write() so each character is creating a separate packet, slowing things down for wifi. You should still use it for serial to save memory.

You may want to use the functions from my post
http://forum.arduino.cc/index.php?topic=161586.msg1208409#msg1208409
to concatinate strings, interger, floats etc. into one string:

char outbuf[512];
...
void appendToOutbuf(String in) {
  char x[256];
  in.toCharArray(x,256);
  appendToOutbuf(x);
}

// As the EthernetClient tends to send each character of e.g. an integer in an own packet, I made
// this function to collect everything to send. Not needed for WiFiClient, but still
// useful, to be on the save side, and to be consistent to EthernetClient-code.
void appendToOutbuf(char in[]) {
  int pos = 0;
  // append-position suchen
  for (pos = 0; pos < outbufLen; pos++) {
    if (outbuf[pos] == '\0') break;
  }
  int pos2 = 0;
  // kopieren
  for (pos2 = 0; pos2 < 200; pos2++) {
    outbuf[pos] = in[pos2];
    if (in[pos2] == '\0') break;
    pos++;
    if (pos == outbufLen) {
      outbuf[pos] = '\0';
      break;
    }
  }
}

The fixed sizes (256 and 512) are really bad, but it does it, and as a non-C-programmer i did my best :wink:

BTW: To verify the amount of sent packets, you may use a ethernet packet sniffer like tcpdump or wireshark. However, maybe the wifi router does packet concatinations on its own, who knows.

Thomas,

I was posting on your thread a question about the maximal length of package you were able to send. For me it was 90 bytes via client.print. Since you have 256 byte buffer, I wonder if there is any occasion where you do need to send that many bytes in one shot and how did it go? For me it never goes. So member pylon looked at the firmware code and found there is a 100 byte buffer in the firmware so nothing beyond 90 will be accepted by the wifi MCU's firmware.

liudr:
Thomas,

I was posting on your thread a question about the maximal length of package you were able to send. For me it was 90 bytes via client.print. Since you have 256 byte buffer, I wonder if there is any occasion where you do need to send that many bytes in one shot and how did it go? For me it never goes. So member pylon looked at the firmware code and found there is a 100 byte buffer in the firmware so nothing beyond 90 will be accepted by the wifi MCU's firmware.

Hi,

This post is a little bit old, but the solution is to make a buffer that will contain all your message to send. Then you have to send it parts by parts and recompose it at the arrival.

Say if you don't understand what I wrote, I will be free to help you :).

Have nice day.
Alessandro

I have just finished some testing of the WiFi Shield's TCP behaviour, using Wireshark.
If you are reading this thread, you may want to read my notes here
http://mssystems.emscom.net/helpdesk/knowledgebase.php?article=51

Maximum throughput is about 11Kbps, which you will get by chunking data into a 90 byte buffer and using write() instead of println().

There are two bottlenecks involved.

  1. The SPI bus and much has been written about the 90 byte limit.
  2. The Shield's transmit cycle, a relatively massive 2 seconds which can not be optimised around.