Why's SD -> Ethernet Card -> Webbrowser so slow?

Hello guys,

Not a real problem, but could somebody explain why data transfer is so terribly slow:

I've an Arduino Ethernet card with 8GB SD card.
This SD card contains a complete html code including several jpg's and png's files.
It's all working fine: the webbrowser gives a request, arduino delivers the html page via the SD card, the webbrowser asks for the images (as put in the html file) and Arduino sent the files.
But an image of approx. 150kB takes about 20 seconds to load, what's the bottleneck around here?
(Arduino, Wiznet 5100, SD card, libraries..)

In all my simplicity :relaxed: I think: 8 MHz arduino, pick a byte from SD, sent it over the internet etc., 150kB shouldn't take 20 seconds?

hi ,
how you make arduino open webpage on sd card having pictures and open it .. i tried to do that for 2 days and failed at the end
any help please

The below discussion may have some useful info and examples.

http://arduino.cc/forum/index.php/topic,144675.0.html

thanks alot really :slight_smile:

If you want to avoid the fate of the OP, then you need to send these files more than one byte per packet. If you are sending a big file, this will take relatively forever.

            while (myFile.available()) {
              client.write(myFile.read());
            }
            // close the file:
            myFile.close();

This sends the data in 64 byte packets, rather than one byte per packet.

  Serial.println("Writing");

  byte clientBuf[64];
  int clientCount = 0;

  while(fh.available())
  {
    clientBuf[clientCount] = fh.read();
    clientCount++;

    if(clientCount > 63)
    {
      Serial.println("Packet");
      client.write(clientBuf,64);
      clientCount = 0;
    }
  }

  if(clientCount > 0) client.write(clientBuf,clientCount);

edit: If you want it to work really fast, remove or comment out the Serial.println("Packet") statement. That was just for debugging.

I put the below in my origional code and it reduced the upload time of a 253k jpg file from 62 sec. to 15 sec.

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            // close the file:
            myFile.close();
          }

@zoomkat: You forgot this part. Without it, it may not send the last packet. There may have been less than 64 bytes in clientBuf.

  if(clientCount > 0) client.write(clientBuf,clientCount);

Compared to 15 seconds, 62 seconds is relatively forever.

Without it, it may not send the last packet. There may have been less than 64 bytes in clientBuf.

Good point, I went and added it back like below. Next project is to get all the file uploading into a single function.

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);            
            // close the file:
            myFile.close();
          }

Why did you choose 64 bytes?? What is the maximum bytes per packet that ethernet shield can send??

blastboot:
Why did you choose 64 bytes?? What is the maximum bytes per packet that ethernet shield can send??

The maximum is about 1400. However, you can't use a variable declaration like this.

byte outBuf[1400];

You don't have that much memory in an Uno. So I wanted to insure it might be able to run on an Uno, but also send more data than header. The header is normally 48 bytes, so I picked 64.

it makes sense :slight_smile: in my code i have

int16_t c;
          while ((c = file.read()) >= 0) {
client.print((char)c);
          }

adapting it with the information you provide, is something like

byte clientBuf[64];
            int clientCount = 0;
              clientBuf[clientCount] = file.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);            
            // close the file:
            file.close();

is this right?

I didn't compile it, but that looks like it! :slight_smile:

Thanks i'll try it because i have the same speed problem with images.
One more thing... what does this line exactly do?

if(clientCount > 0) client.write(clientBuf,clientCount);

That sends the last packet. Unless you are really lucky, your files will not be an even multiple of 64, so there may be a few bytes remaining in that buffer when the file read is finished.

got it :slight_smile: thanks!!

Can i use this system to print html commands like we do with client.print?

Yes. It doesn't matter whether client or server. Any time you use a tcp protocol, this will apply.

thanks again! 8)

I've tested this with my code, but for some reason it didn't work like expected, and it was slower than my original code, but maybe i've made something wrong

#include <SPI.h>
#include <SdFat.h>
#include <SdFatUtil.h>
#include <Ethernet.h>
#include <Flash.h>
#include <VirtualWire.h>;
#define ledIndicadorComms 2
#define configConcl 3
#define ledRececao 7

#define BUFSIZ 128
const int pinoRececao = 8;
int leitura = 0;
float percentagem = 0;
float litros = 0;
float dados = 0;
char mensagem[5];
boolean verifica = false;
boolean verifica2 = false;
boolean Log = false;
int nr = 1;

byte mac[] = { 
  0x90, 0xA2, 0xDA, 0x00, 0xEA, 0x8C };
IPAddress ip(196,168,0,3);
char rootFileName[] = "index.htm";
char ficheiroLog[] ="&niv_LOG.txt";
EthernetServer server(80);

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
SdFile LOGfile;

#define error(s) error_P(PSTR(s))
void error_P(const char* str) {
  PgmPrint("Erro: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("Erro cartao SD: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  digitalWrite(configConcl, HIGH);
  while(1);
}

void setup() {

  Serial.begin(9600);
  delay(1);
  pinMode(ledIndicadorComms, OUTPUT);
  pinMode(configConcl, OUTPUT);
 
  PgmPrint("Memoria RAM Livre: ");
  Serial.println(FreeRam());  
    
  pinMode(10, OUTPUT);              
  digitalWrite(10, HIGH);              

  if (!card.init(SPI_FULL_SPEED, 4)) error("Falha no card.init!");

  if (!volume.init(&card)) error("Falha no vol.init!");

  PgmPrint("O volume e FAT");
  Serial.println(volume.fatType(),DEC);
  Serial.println();

  if (!root.openRoot(&volume)) error("Falha ao abrir raiz do cartao!");

  PgmPrintln("Ficheiros na raiz do cartao:");

  root.ls(LS_DATE | LS_SIZE);
  Serial.println();

  PgmPrintln("Ficheiros encontrados em todos os directorios do cartao:");
  root.ls(LS_R);

  Serial.println();
  PgmPrintln("Concluido");

  Log = file.open(&root, ficheiroLog, O_WRITE | O_APPEND);
  if (Log){
    file.println("Leitura  ,  Nivel Tanque  ,  Quantidade Agua");
    file.close();
    Serial.println(F("Escrita do Header no ficheiro LOG concluida!"));
  }else{
    Serial.println(F("ERRO! Nao foi possivel abrir o ficheiro LOG."));

    while(1){
      digitalWrite(configConcl, HIGH);
      delay(1000);
      digitalWrite(configConcl,LOW);
      delay(1000);
    }  
}    

  Ethernet.begin(mac, ip);
  server.begin();
  delay(2);

  pinMode(ledRececao, OUTPUT);
  vw_set_rx_pin(pinoRececao);
  vw_setup(2000);
  vw_rx_start();

  digitalWrite(configConcl, HIGH);
  delay(250);
  digitalWrite(configConcl,LOW);
  delay(250);
  digitalWrite(configConcl, HIGH);
  delay(250);
  digitalWrite(configConcl,LOW);
}


void loop() {
  char clientline[BUFSIZ];
  char *filename;
  int image = 0;
  int index = 0;
  verifica = false;
  Log = false;

uint8_t buf[VW_MAX_MESSAGE_LEN]; 
uint8_t buflen = VW_MAX_MESSAGE_LEN; 

  if (vw_get_message(buf, &buflen)){
    int i;
    for (i=0; i<buflen; i++){ 
    mensagem[i] = buf[i];
    
    if (mensagem[0] == 'R'){
      leitura = atoi(&mensagem[1]);
      percentagem = (leitura * 100)/465;
      litros = (percentagem*700000)/100;

  Serial.print(F("Nivel do Tanque:"));
  Serial.print(percentagem, 1);
  Serial.println(F("%"));          
  Serial.print(F("Quantidade de agua no Tanque:"));
  Serial.print(litros, 1);
  Serial.println(F("litros"));

    Log = file.open(&root, ficheiroLog, O_WRITE | O_APPEND);
    if (Log){
    file.print(nr);
    file.print("  ,  ");
    file.print(percentagem, 1);
    file.print("  ,  ");
    file.println(litros, 1);
    file.close();
    
    Serial.println(F("Valores gravados com sucesso!"));
    nr++;
    Log = false;
    } else {
     Serial.println(F("Nao foi possivel gravar dados no ficheiro LOG!"));
    } 
  }
  memset(&mensagem, 0, sizeof(mensagem)); //APAGA A MENSAGEM RECEBIDA
  }
  }

  EthernetClient client = server.available();
  if (client) {
    boolean current_line_is_blank = true;

    index = 0;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        if (c != '\n' && c != '\r') {
          clientline[index] = c;
          index++;

          if (index >= BUFSIZ) 
            index = BUFSIZ -1;
            continue;
        }

        clientline[index] = 0;
        filename = 0;

        Serial.println(clientline);

        digitalWrite(ledIndicadorComms, HIGH);
        delay(1);
        digitalWrite(ledIndicadorComms, LOW);
        delay(1);

        if (strstr(clientline, "GET / ") != 0) {
          filename = rootFileName;
          } 

        if (strstr(clientline, "GET /") != 0) {

          if (strstr(clientline, "GET /iframe") != 0) {
            verifica = true;
          }
 
          if (strstr(clientline, "GET /2iframe") != 0) {
            verifica2 = true;
          }
          if(strstr(clientline, "GET/&niv") !=0){
            break;
          }
          if (!filename) filename = clientline + 5; 

          (strstr(clientline, " HTTP"))[0] = 0;

          Serial.println(filename);

          if (! file.open(&root, filename, O_READ)) {
            client.println(F("HTTP/1.1 404 Not Found"));
            client.println(F("Content-Type: text/html"));
            client.println();
            client.println(F("<h2>Erro 404</h2>"));
            client.println(F("<s2>O ficheiro nao existe.<s2>"));
            client.println(F(""));
            break;
          }
          Serial.println("Ligado!");
client.println("HTTP/1.1 200 OK");
          if (strstr(filename, ".htm") != 0)
            client.println("Content-Type: text/html");
          else if (strstr(filename, ".css") != 0)
            client.println("Content-Type: text/css");
          else if (strstr(filename, ".png") != 0)
            client.println("Content-Type: text/php");
          else if (strstr(filename, ".php") != 0)
            client.println("Content-Type: image/png");
          else if (strstr(filename, ".jpg") != 0)
            client.println("Content-Type: image/jpeg");
          else if (strstr(filename, ".gif") != 0)
            client.println("Content-Type: image/gif");
          else if (strstr(filename, ".3gp") != 0)
            client.println("Content-Type: video/mpeg");
          else if (strstr(filename, ".pdf") != 0)
            client.println("Content-Type: application/pdf");
          else if (strstr(filename, ".js") != 0)
            client.println("Content-Type: application/x-javascript");
          else if (strstr(filename, ".xml") != 0)
            client.println("Content-Type: application/xml");
          else
            client.println("Content-Type: text");
            client.println();

            byte clientBuf[64];
            int clientCount = 0;
           while(file.available()){
              clientBuf[clientCount] = file.read();
              clientCount++;
              if(clientCount > 63) {
                Serial.println("Pacote de dados de 64 bytes enviado!");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
              if(clientCount > 0) client.write(clientBuf,clientCount);
            
           client.print("<html><body>");

          if (verifica){
          client.println(F("<div id=\"outer\">"));
          client.print(F("<div id=\"inner\" style=\"height:"));
          client.print(percentagem, 1);
          client.println(F("%\">"));
          client.println(F("</div></div>"));
          client.println("</p></body></html>");
          }   
          if (verifica2){
           if (percentagem >= 90 || percentagem <=10){
           client.println(F("<table width=\"210\" height=\"42\" border=\"0\" cellpadding=\"1\" cellspacing=\"1\">"));
           client.println(F("<tr>"));
           client.print(F(" <td width=\"69\" height=\"40\" class=\"iframe2\"><div align=\"center\"><b><font color=#E60000>"));
           client.print(percentagem, 1);
           client.println(F("%</font></b></div></td>"));
           client.print(F("<td width=\"134\" class=\"iframe2\"><div align=\"center\"><b><font color=#E60000>"));
           client.print(litros, 1);
           client.print(F("</font></b></div></td>"));
           client.println(F("</tr>"));
           client.println(F("</table>"));
           client.println("</p></body></html>");
           } else {
           client.println(F("<table width=\"210\" height=\"42\" border=\"0\" cellpadding=\"1\" cellspacing=\"1\">"));
           client.println(F("<tr>"));
           client.print(F(" <td width=\"69\" height=\"40\" class=\"iframe2\"><div align=\"center\"><b>"));
           client.print(percentagem, 1);
           client.println(F("%</b></div></td>"));
           client.print(F("<td width=\"134\" class=\"iframe2\"><div align=\"center\"><b>"));
           client.print(litros, 1);
           client.print(F("</b></div></td>"));
           client.println(F("</tr>"));
           client.println(F("</table>"));
           client.println("</p></body></html>");
       }
          }
          verifica = false;
          verifica2 = false;
          file.close();

        }
        else {
          client.println("HTTP/1.1 404 Not Found");
          client.println("Content-Type: text/html");
          client.println();
          client.println("<h2>Error 404</h2>");
          client.println("");
        }
        break;
      }
          if(strstr(clientline, "GET/&niv") !=0){
            Log = file.open(&root, ficheiroLog, O_READ | O_APPEND);
            if (Log){
              Serial.println(F("A enviar ficheiro LOG para download!"));
              byte clientBuffer[64];
              int clientCounter = 0;
                while(file.available()) {
                clientBuffer[clientCounter] = file.read();
                clientCounter++;
                if(clientCounter > 63) {
                Serial.println("Pacote de dados de 64 bytes enviado!");
                client.println("Content-Disposition: attachment; filename=\"ficheiroLOG.txt\"");
                client.write(clientBuffer,64);
                clientCounter = 0;
              }
            }
              if(clientCounter > 0) client.write(clientBuffer,clientCounter);
          file.close();
      } else {
        client.println("HTTP/1.1 404 Not Found");
        client.println("Content-Type: text/html");
        client.println();
        client.println("<h2>ERRO! Ficheiro LOG não existe!</h2>");
        client.println("");
    }
    }     
  }

//PISCA LED PARA INDICAR QUE ESTÁ CONCLUÍDA A COMUNICAÇÃO SERVIDOR-CLIENTE E CLIENTE-SERVIDOR
    digitalWrite(ledIndicadorComms, HIGH);
    delay(1);
    digitalWrite(ledIndicadorComms, LOW);
    delay(1);
    client.stop();
  }
}

/*FIM DO PROGRAMA*/

Where is

//COLOCA EM BUFFER OS BYTES LIDOS DO FICHEIRO DO CARTÃO SD
            byte clientBuf[64];
            int clientCount = 0;
           while(file.available()){
              clientBuf[clientCount] = file.read();
              clientCount++;
              if(clientCount > 63) {
                Serial.println("Pacote de dados de 64 bytes enviado!");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
              if(clientCount > 0) client.write(clientBuf,clientCount);

I had on my original code

 int16_t c;
          while ((c = file.read()) >= 0) {
            client.print((char)c);
          }