Web server refresh problem

          print(F("<input type=button value=F0 style=position:absolute;left:775px;top:150px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F0'>&nbsp;&nbsp;"));
          print(F("<input type=button value=F1 style=position:absolute;left:775px;top:230px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F1'>&nbsp;&nbsp;"));
          print(F("<input type=button value=F2 style=position:absolute;left:775px;top:310px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F2'>&nbsp;&nbsp;"));
          print(F("<input type=button value=F3 style=position:absolute;left:775px;top:390px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F3'>&nbsp;&nbsp;"));
          print(F("<input type=button value=F4 style=position:absolute;left:775px;top:470px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F4'>&nbsp;&nbsp;"));
          print(F("<input type=button value=F5 style=position:absolute;left:775px;top:550px;width:60px;height:50px;color:DarkRed;font-weight:bold;font-size;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href='/?F5'>&nbsp;&nbsp;"));

Have you considered writing a function to output this? Try to remember that you have limited RAM and program memory. You have the words "color:DarkRed;font-weight:bold;font-size:16px;border-width:5px;border-style:ridge;border-color:DarkRed; onmousedown=location.href" repeated many times. Write a function that takes the variable parts, and outputs the line for you. Also the code would be much easier to read.


Use an UNO board and try somehow to fit it on an UNO?

There's no "somehow" about it. The code is incredibly inefficiently written.

As long as you are using the default Ethernet library - you will never be able to make it fast.

Ethernet library that comes with Arduino is flawed and has errors in it, that make it so that the server 'hangs' for 5-7 seconds. (AFAIK something to do with not picking the right socket)

To save more space - you can also use a CSS class, so you son't have to repeat your styles all the time.
And put all this in there:

input[type="button"] {
  position:absolute;
  width:60px;
  height:50px;
  color:DarkRed;
  font-weight:bold;
  font-size:16px;
  border-width:5px;
  border-style:ridge;
  border-color:DarkRed;  /* you dont really need this, if left unset 'border-color' will be same as 'color' */
}

The advantage of that would be to actually lower the time taken to send, where making a function still sends the same stuff. So YemSalat's suggestion is the better one.

However you could still make a function to draw the buttons, using the CSS class.

YemSalat:
As long as you are using the default Ethernet library - you will never be able to make it fast.

You will not make it fast like a Apache server running on a PC, but I can make it pretty darned fast. Read reply #9. Nick Gammon cut the download time to 1/5 the original time.

YemSalat:
Ethernet library that comes with Arduino is flawed and has errors in it, that make it so that the server 'hangs' for 5-7 seconds. (AFAIK something to do with not picking the right socket)

Can you show me your code that does that? I have never had any of my servers hang for 5 to 7 seconds. Mine picks an available socket every time, unless there is not one available.

I've made my server code about as bulletproof as I can. I intentionally torture it with things that would crash about any other server example around. Give it a try. See if you can crash it.
http://playground.arduino.cc/Code/WebServerST
If you enter an 'r' on the serial monitor and press the enter key, it will display the status for all sockets.

SurferTim:
You will not make it fast like a Apache server running on a PC

I'll grant you that.

SurferTim:
Can you show me your code that does that? I have never had any of my servers hang for 5 to 7 seconds. Mine picks an available socket every time, unless there is not one available.

Perhaps I should just quote the library code from EthernetClient.cpp:

EthernetClient EthernetServer::available() {
  accept();

  for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
    EthernetClient client(sock);
    if (EthernetClass::_server_port[sock] == _port &&
        (client.status() == SnSR::ESTABLISHED ||
         client.status() == SnSR::CLOSE_WAIT)) {
      if (client.available()) {
        // XXX: don't always pick the lowest numbered socket.
        return client;
      }
    }
  }
  return EthernetClient(MAX_SOCK_NUM);
}

Notice this?
// XXX: don't always pick the lowest numbered socket

Here is how this can be fixed:

EthernetClient EthernetServer::available_(int sock) {
  accept_(sock);
  EthernetClient client(sock);
  if (EthernetClass::_server_port[sock] == _port &&
      (client.status() == SnSR::ESTABLISHED ||
       client.status() == SnSR::CLOSE_WAIT)) {
    if (client.available()) {
      return client;
    }
  }
  return EthernetClient(MAX_SOCK_NUM);
}

And also need to redefine the accept function:

void EthernetServer::accept_(int sock) {
  int listening = 0;
  EthernetClient client(sock);

  if (EthernetClass::_server_port[sock] == _port) {
    if (client.status() == SnSR::LISTEN) {
      listening = 1;
    } 
    else if (client.status() == SnSR::CLOSE_WAIT && !client.available()) {
      client.stop();
    }
  } 

  if (!listening) {
    //begin();
    begin_(sock); // added
  }
}

Also, update EthernetServer.h to use the 'redefined' functions:

class EthernetServer : 
public Server {
private:
  uint16_t _port;
  //void accept();
  void accept_(int sock);
public:
  EthernetServer(uint16_t);
  //EthernetClient available();
  EthernetClient available_(int sock);
  virtual void begin();
  virtual void begin_(int sock);
  virtual size_t write(uint8_t);
  virtual size_t write(const uint8_t *buf, size_t size);
  using Print::write;
};

NOTE: this solution is not mine, the original solution was posted by @smart_alex about a week ago, but I don't think it is available in English, so just reposting here in case others find it useful.

In his test he got the following results (I was also experimenting with Ethernet for the past couple days and notice delays as well)

PS:
мс = ms = milliseconds
с = s = seconds

Before modifications to the library:

After

As you may see it is still not perfect (3s delay on style.css) but still much better then original.

Complete modified EthernetServer.cpp:

/*
Mod for Arduino Mega Server project
fix bug delay answer server
*/

#include "w5100.h"
#include "socket.h"
extern "C" {
#include "string.h"
}

#include "Ethernet.h"
#include "EthernetClient.h"
#include "EthernetServer.h"

EthernetServer::EthernetServer(uint16_t port) {
_port = port;
}

void EthernetServer::begin() {
for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
EthernetClient client(sock);
if (client.status() == SnSR::CLOSED) {
socket(sock, SnMR::TCP, _port, 0);
listen(sock);
EthernetClass::_server_port[sock] = _port;
break;
}
}
}

void EthernetServer::begin_(int sock) {
EthernetClient client(sock);
if (client.status() == SnSR::CLOSED) {
socket(sock, SnMR::TCP, _port, 0);
listen(sock);
EthernetClass::_server_port[sock] = _port;
}
}

/*

void EthernetServer::accept() {
int listening = 0;

for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
EthernetClient client(sock);

if (EthernetClass::_server_port[sock] == _port) {
if (client.status() == SnSR::LISTEN) {
listening = 1;
}
else if (client.status() == SnSR::CLOSE_WAIT && !client.available()) {
client.stop();
}
}
}

if (!listening) {
begin();
}
}

*/

void EthernetServer::accept_(int sock) {
int listening = 0;
EthernetClient client(sock);

if (EthernetClass::_server_port[sock] == _port) {
if (client.status() == SnSR::LISTEN) {
listening = 1;
}
else if (client.status() == SnSR::CLOSE_WAIT && !client.available()) {
client.stop();
}
}

if (!listening) {
//begin();
begin_(sock); // added
}
}

/*

EthernetClient EthernetServer::available() {
accept();

for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
EthernetClient client(sock);
if (EthernetClass::_server_port[sock] == _port &&
(client.status() == SnSR::ESTABLISHED ||
client.status() == SnSR::CLOSE_WAIT)) {
if (client.available()) {
// XXX: don't always pick the lowest numbered socket.
return client;
}
}
}
return EthernetClient(MAX_SOCK_NUM);
}

*/

EthernetClient EthernetServer::available_(int sock) {
accept_(sock);
EthernetClient client(sock);
if (EthernetClass::_server_port[sock] == _port &&
(client.status() == SnSR::ESTABLISHED ||
client.status() == SnSR::CLOSE_WAIT)) {
if (client.available()) {
return client;
}
}
return EthernetClient(MAX_SOCK_NUM);
}

size_t EthernetServer::write(uint8_t b) {
return write(&b, 1);
}

size_t EthernetServer::write(const uint8_t *buffer, size_t size) {
size_t n = 0;
//accept();

for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
accept_(sock); // added
EthernetClient client(sock);

if (EthernetClass::_server_port[sock] == _port &&
client.status() == SnSR::ESTABLISHED) {
n += client.write(buffer, size);
}
}
return n;
}

EthernetServer.h

/*
Mod for Arduino Mega Server project
fix bug delay answer server
*/

#ifndef ethernetserver_h
#define ethernetserver_h

#include «Server.h»

class EthernetClient;

class EthernetServer:
public Server {
private:
uint16_t _port;
//void accept();
void accept_(int sock);
public:
EthernetServer(uint16_t);
//EthernetClient available();
EthernetClient available_(int sock);
virtual void begin();
virtual void begin_(int sock);
virtual size_t write(uint8_t);
virtual size_t write(const uint8_t *buf, size_t size);
using Print::write;
};

#endif

[Edit] made images 'clickable'
[Edit 2] added complete modified versions of EthernetServer, all credit to @smart_alex

Forgot to add, since the library code was updated, its methods should be called in a slightly different way:
Now since the code is updated - you can just cycle through sockets 'manually' and pick the right one:

 for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
    EthernetClient sclient = server.available_(sock);
    serverWorks(sclient);
  }

Your function:

void serverWorks(EthernetClient sclient) {

}

YemSalat:
Perhaps I should just quote the library code from EthernetClient.cpp:

EthernetClient EthernetServer::available() {

accept();

for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
    EthernetClient client(sock);
    if (EthernetClass::_server_port[sock] == _port &&
        (client.status() == SnSR::ESTABLISHED ||
        client.status() == SnSR::CLOSE_WAIT)) {
      if (client.available()) {
        // XXX: don't always pick the lowest numbered socket.
        return client;
      }
    }
  }
  return EthernetClient(MAX_SOCK_NUM);
}



Notice this?
// XXX: don't always pick the lowest numbered socket

And that causes a problem how? edit: It picks the lowest socket that is active and has characters available in the rx queue.

As you may see it is still not perfect (3s delay on style.css) but still much better then original.

I don't see that delay unless the server must serve a really large html file that takes 3 seconds to download.

I get that kind of crappy performance if I don't send the files in large packets. That was the point I was making.

SurferTim:
And that causes a problem how? edit: It picks the lowest socket that is active and has characters available in the rx queue.

It cycles through sockets and if it finds one - it will return the client and ignore the rest.

The solution that I posted is rather 'brute force' and I don't like that it transfers part of the library logic to the code.

To be honest I personally do not completely understand what the problem is yet. I've been looking for various explanations for the past couple days. I thought that the problem is sockets not closing properly, but that does not seem to be the issue.
I don't know why but sometimes the socket will just hang for 3 seconds (default TCP timeout?)

There is a discussion realted to this here: Accettare più connessioni in ingresso su server ethernet - Software - Arduino Forum (its in Italian)

I don't see that delay unless the server must serve a really large html file that takes 3 seconds to download.

I get that kind of crappy performance if I don't send the files in large packets. That was the point I was making.

Can you please post a screenshot of your browser devtools network tab when loading a web page with some files in it?

YemSalat:
Complete modified EthernetServer.cpp:

#include «w5100.h»

#include «socket.h»
extern «C» {
...

Your posted code has some weird quotes in it.

My personal experience has been that the official Ethernet card works fine, and a cheap eBay clone keeps displaying the symptoms of not connecting more often than not. In fact, altering the library as suggested did not help in this case.

So I would say: "you get what you pay for".

YemSalat:
Forgot to add, since the library code was updated, its methods should be called in a slightly different way:
Now since the code is updated - you can just cycle through sockets 'manually' and pick the right one:

 for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {

EthernetClient sclient = server.available_(sock);
    serverWorks(sclient);
  }

I tried:

 EthernetClient client;
 for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
    client = server.available_(sock);
    if (client)
      break;
  }

  if (!client)
    return;

I'm not sure if that is OK or not.

Oops, sorry, was copy-pasting from an online code highlighter, not sure why it changed the quotes (will fix in a second)

I will post my own code that I did tests with later (not on my laptop at the moment)

Could be the case, I got mine on ebay for about $6-7

YemSalat:
I thought that the problem is sockets not closing properly, but that does not seem to be the issue.

That is exactly what I found. Sockets not closing properly.

YemSalat:
Can you please post a screenshot of your browser devtools network tab when loading a web page with some files in it?

Why? My serial debug display shows the request and response to every file the web browser requests. That includes html, css, javascript, and two images (one jpg and the other favicon.ico) all on one page. Not only that, but the web browser displays the page exactly as it should.

SurferTim:
Why? My serial debug display shows the request and response to every file the web browser requests. That includes html, css, javascript, and two images (one jpg and the other favicon.ico) all on one page.

Browser devtools will give a better visual representation of how requests are fired / executed.

I think on both Chrome and Firefox the shortcut to launch devtools is Ctrl+Shift+i
Then you can just switch to the 'Network' tab and reload the page.

Not only that, but the web browser displays the page exactly as it should.

Why wouldn't it? Web browsers don't care at what speed / order they get the files as long as they get all of them in the end. We are talking about delays here.

Some simple server iframe web page test code that shows the arduino server can supply small pages fairly fast.

// zoomkat's meta refresh data frame test page 5/25/13
// use http://192.168.1.102:84 in your brouser for main page
// http://192.168.1.102:84/data static data page
// http://192.168.1.102:84/datastart meta refresh data page
// for use with W5100 based ethernet shields
// set the refresh rate to 0 for fastest update
// use STOP for single data updates

#include <SPI.h>
#include <Ethernet.h>

const int analogInPin0 = A0;
const int analogInPin1 = A1;
const int analogInPin2 = A2;
const int analogInPin3 = A3;
const int analogInPin4 = A4;
const int analogInPin5 = A5;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // arduino ip in lan
byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
unsigned long int x=0; //set refresh counter to 0
String readString; 

//////////////////////

void setup(){
  Serial.begin(9600);
    // disable SD SPI if memory card in the uSD slot
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  server.begin();
  Serial.println("meta refresh data frame test 5/25/13"); // so I can keep track of what is loaded
}

void loop(){
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
         if (readString.length() < 100) {
          readString += c; 
         } 
        //check if HTTP request has ended
        if (c == '\n') {

          //check get atring received
          Serial.println(readString);

          //output HTML data header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();

          //generate data page
          if(readString.indexOf("data") >0) {  //checks for "data" page
            x=x+1; //page upload counter
            client.print("<HTML><HEAD>");
            //meta-refresh page every 1 seconds if "datastart" page
            if(readString.indexOf("datastart") >0) client.print("<meta http-equiv='refresh' content='1'>"); 
            //meta-refresh 0 for fast data
            if(readString.indexOf("datafast") >0) client.print("<meta http-equiv='refresh' content='0'>"); 
            client.print("<title>Zoomkat's meta-refresh test</title></head><BODY>
");
            client.print("page refresh number: ");
            client.print(x); //current refresh count
            client.print("

");
            
              //output the value of each analog input pin
            client.print("analog input0 is: ");
            client.print(analogRead(analogInPin0));
            
            client.print("
analog input1 is: ");
            client.print(analogRead(analogInPin1));
                        
            client.print("
analog input2 is: ");
            client.print(analogRead(analogInPin2));
            
            client.print("
analog input3 is: ");
            client.print(analogRead(analogInPin3));
                                    
            client.print("
analog input4 is: ");
            client.print(analogRead(analogInPin4));
            
            client.print("
analog input5 is: ");
            client.print(analogRead(analogInPin5));
            client.println("
</BODY></HTML>");
           }
          //generate main page with iframe
          else
          {
            client.print("<HTML><HEAD><TITLE>Zoomkat's frame refresh test</TITLE></HEAD>");
            client.print("Zoomkat's Arduino frame meta refresh test 5/25/13");
            client.print("

Arduino analog input data frame:
");
            client.print("&nbsp;&nbsp;<a href='http://192.168.1.102:84/datastart' target='DataBox' title=''yy''>META-REFRESH</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/data' target='DataBox' title=''xx''>SINGLE-STOP</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/datafast' target='DataBox' title=''zz''>FAST-DATA</a>
");
            client.print("<iframe src='http://192.168.1.102:84/data' width='350' height='250' name='DataBox'>");
            client.print("</iframe>
</HTML>");
          }
          delay(1);
          //stopping client
          client.stop();
          //clearing string for next read
          readString="";
        }
      }
    }
  }
}

Can you please post a screenshot of your browser devtools network tab when you load that page on your arduino? (I posted instructions on how to do that in my previous comment)

YemSalat:
Can you please post a screenshot of your browser devtools network tab when you load that page on your arduino? (I posted instructions on how to do that in my previous comment)

I've tried zoomkat's code with Arduino 1.6.5 and the default Ethernet library and could not find any problems with time gaps in the handling of HTTP requests.

If the webserver gets under heavy stress and cannot establish every connection that is requested, the webbrowser (Google Chrome) seems to create a priority scheduling for the files requested.

In case of zoomkat's code, the webbrowser will always try to retrieve two files at highest possible speed:

  • datafast
  • favicon.ico

But it seems to be the webbrowser that delays requests to favicon.ico

In the picture included you can see: If the webbrowser is able to establish a connection to the Arduino webserver, it will always try to handle 'datafast' in shortest possible time.

When scrolling down (not visible in the image) I can see, that the response time to 'datafast' always stay short, while the response time to 'favicon.ico' is increasing more and more: 3s, 6s. 9s, 12s, up to more than 30 seconds!

So for me its the webbrowser that creates delays for certain files, possibly the decision is made by file extension. Or 'favicon.ico' has a low priority by default, because it is a special file that the browser requests.

The gap in the timeline chart when 'favicon.ico' is delayed for 3 seconds for the first time, is not existing. When scrolling down the timeline chart I can see that all the time is used: It is used to fulfil 'datafast' requests, which seem to have a higher priority to the Google Chrome webbrowser, while the webserver cannot fulfil the requests fast enough.

I cannot detect a problem with the Ethernet library and the Arduino webserver seems to handle requests all the time, without any gaps in the timeline.

Thanks for the test, but it is not very representative, there should really be more files in there in order to test it properly (sorry I should have looked through zoomkat's code better)

It is also better if the page is loaded just once, not constantly refreshed, so it doesn't affect the results. (And disable cahce of course)

I have not noticed difference when testing between Firefox/Chrome.