Q/A and some unanswered questions for WiFi and web server topics

I am just starting up on an Arduino WiFi server endeavor. I wonder if there are some resources I can read or borrow code from in order to construct my own web server with forms. I know this one but it is too simple:

http://playground.arduino.cc/Code/WebServer

I am using forms as a web interface for my Arduino data collection system. I wonder if there are any resources I can read to understand this process and avoid common pitfalls. Here is my share of experience for anyone interested in reading:

I met a few difficulties while expanding from serving a simple page to serving forms and getting parameter and value from the forms. Here are some:

  1. I am using firefox as client. If I have an input field and there is a space before the value -2.5 like “(space)-2.5” then firefox sends value=±2.5, which messes up the server (weak error handling). I eventually added code to first limit the length of input and then strip off any + before converting to number and strip off any space before embedding the value to the web page. This has stopped some problems. Update: the true answer is URI encoding. Since the parameters are put on URI with form GET command, values have to be a-z, 0-9, -, and a few other symbols only. All other symbols need to be encoded, such as (space) replaces ‘+’ etc.
    Percent-encoding - Wikipedia

  2. I also have some trouble keeping track where all the double quotations are. I now use firefow->view source to check for mistakes with the web page my arduino server sends. It also might be beneficial to use sprintf so any HTML tags with embedded values will be a complete message instead of several pieces:

char buffer[128];
sprintf(buffer, "Set temperature: <input type=\"text\" name=\"Temp\" value=\"%d\" maxlength=\"%d\">", temperature, maxlength)
client.println(buffer);

Instead of:

client.print("Set temperature: <input type=\"text\" name=\"Temp\" value=\"");
client.print(temperature);
client.print("\" maxlength=\"");
client.print(maxlength);
client.println("\">");
client.println(buffer);
  1. Make sure each client.print prints a string not longer than around 90 bytes. Otherwise the string is not sent to the client. Here is my other post about this topic:
    http://arduino.cc/forum/index.php/topic,161642.0.html
    My WiFi shield version is R3 and I have not updated its firmware.

  2. WiFi shield renewing IP address - I read my wireless router’s log. It seems like the WiFi shield is renewing its IP address every few hours (I could be wrong). If you have a wireless router, read its log. I’m more than glad to hear from you regarding IP address renewals, which is essential to staying connected for long periods of time.

  3. “wifi shield not present” - I have some trouble starting the wifi shield as others pointed out but my programming jumper is already removed. I had to reset multiple times and couldn’t pin down how this happened. I suspect it is timing issue, how long arduino takes to reset and how long wifi takes to reset. I’ll try some delays and update this post if I find a pattern of fail to find wifi shield and how to fix it.
    Here is some tentative solution to “wifi shield not present” problem: Instead of if (wWiFi.status()==Wl_NO-SHIELD), use a time out with delay to test the shield for up to 30 seconds:

  unsigned long start=millis();
  while (WiFi.status() == WL_NO_SHIELD) 
  {
    if ((millis()-start)>30000)
    {
      Serial.println("WiFi shield not present"); 
      // don't continue:
      while(true);
    }
    delay(500);
  }

This seems to have worked every single time (no more problems).

  1. WiFi file transfer example and speed too slow problem - I tested this with transferring text file from SD card on WiFi shield to PC and mobile devices with Arduino acting as web server. Initially I used client.write() to write a char that I read from SD file. It was very slow, 70-80Byte/s. I then improved to up to 15KB/s with client.write(char* buffer, int lenght), which is not documented by Arduino team. I also used an undocumented function to read SD card, File.read(char* buffer,int length). This goes very well with the client.write(char*,int). Code here:
    client->println("HTTP/1.1 200 OK");
    client->println("Content-Type: text/csv");
    client->print("Content-Length: ");
    client->println(myFile.size());
    client->println("Connnection: close");
    client->println();
    while(myFile.available())
    {
      int n=myFile.read(buffer,max_wifi_package_length); // multiple byte read.
      if (client->connected())
      {
        client->write((uint8_t*)buffer,n); // Only send if the client is still connected.
      }
      else
      {
        Serial.println("Client disconnected before file complete.");
        break;
      }
    }
    client->println();
    myFile.close();

There are several features such as sensing whether the connection is still active, and telling client the size of the file, both of which probably helped arduino and the client. Related post below:

To know more about how these write and print are related, here is a related post:

More to come, some questions with answers and waiting for answers.

A couple of points:

  • the “+” is not an error. GET/POST data has a well-defined character set. Anything outside the set of allowed characters is converted to an allowed representation – e.g., a becomes a ‘+’, and ‘+’ becomes %2B (ASCII code for ‘+’ represented as a hexadecimal value).

You may use apostrophe OR double-quote to delimit strings sent to the browser. So you can enclose attributes in apostrophes while the string containing your HTML tag can be delimited with the double-quote:

client.print(" ‘maxlength=’ "); // Extra white-space added for clarity of explanation

The rules for correctly-formed http messages are well documented in laborious-to-read RFC specifications like RFC-2616. You can probably figure out what you need to do by using Firefox with something like the HTTPfox addin. The RFC is well-suited to curing insomniacs.

Thanks for pointing out the encoding John! I found this short passage on wiki:

http://en.wikipedia.org/wiki/Query_string#URL_encoding

I have the RFC2616 in my dropbox :)

By the way, I POST some data to a webserver in the following manner. Note the two ‘replace’ lines. My quick and dirty url encoding that satisfied my particular needs although it’s not a general solution:

void AlarmLogger::_serverPostMessage2(radiopacket_t data, String sTime) {

  String PostData = "message=";
  PostData += data.payload.type2.message;
  PostData += "&sensor=";
  PostData += data.sensor_id;
  PostData += "&event=";
  PostData += data.payload.type2.message_code;
  PostData += "&time=";
  PostData += sTime;

  Serial.println(PostData);
  
  PostData.replace('\0',' ');
  PostData.replace(' ', '+');
  
  if (_client.connect(_logger_ip, _logger_port)) {

    Serial.print(F("Connecting..."));

    _client.print(  F("POST /logger.php"));
    _client.println(F(" HTTP/1.1"));
    _client.println(F("Host: 192.168.0.9"));
    _client.println(F("User-Agent: Arduino/1.0"));
    _client.println(F("Content-Type: application/x-www-form-urlencoded"));
    _client.println(F("Connection: close"));
    _client.print(  F("Content-Length: "));
    _client.println(PostData.length());
    _client.println();
    _client.println(PostData);
    _client.stop();

    Serial.println(F("Posted"));
  }
  else {
    Serial.println(F("Did not connect"));
  }
}

Thank you John! Are you using Ethernet or WiFi? If you are using Arduino WiFi shield, did you notice the bullet point 3 about max length of client.print()?

Just using Ethernet here.

Solution to question 5 (tentative) has been added to OP. I also posted a comparison between client.write() and client.print as question 6.

"WiFi shield not present" problem (5) has been permanently resolved. One just has to keep checking on the shield instead of checking it one time immediately after startup like the arduino sample code does. The problem is updated.