Pages: [1]   Go Down
Author Topic: SD.h library reading speed - largely WiFi shield issue - ongoing  (Read 3185 times)
0 Members and 1 Guest are viewing this topic.
Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

[Edit] Largely the issue is WiFi shield related although readers interested in SD library can also learn a few things from this ongoing thread.

I am using the SD library included in Arduino 1.0 with Arduino MEGA and Arduino WiFi shield for an Arduino web server test.

I admit I should have learned the sdfat library instead but things have already been developed with SD library so maybe next project. I noticed when using Arduino as a web server, the file download speed is pretty slow. My Arduino web server is able to send about 75 bytes per second to a local network host. All I was doing is a while(myFile.available()) loop with client.write(myFile.read());

I was wondering if SD library is to be blamed for the slow speed. If you use a class 4 micro-SD card with SD library, what kind of reading speed are you expected to see? Is there a read function that reads multiple bytes (blocks) into the memory instead of reading one byte at a time?

Thank you!
« Last Edit: May 04, 2013, 02:45:28 pm by liudr » Logged


SE USA
Offline Offline
Faraday Member
**
Karma: 41
Posts: 3783
@ssh0le
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

the stock sd library is a watered down version of sdfatlib, and you can get that going pretty fast if you use a 74hc4050 for level shifting (dont have an exact number off the top of my head) guess it depends on which SPI speed its set to in the library
« Last Edit: May 03, 2013, 12:29:05 am by Osgeld » Logged


0
Offline Offline
Edison Member
*
Karma: 67
Posts: 1656
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I don't think SD.h is the problem.  Single byte reads are slow but you should be able to read several KB/sec.

I suspect the single byte write to the network is the main problem.

Using the SD.h function
Code:
int read(void *buf, uint16_t nbyte);
helps read speed a lot.  You should be able to read 50-100 KB/sec with a 10 byte buffer and well over 100 KB/sec with a larger buffer.

Try a multi-byte read/write something like this:
Code:
  uint8_t buf[BUF_DIM];
  while(myFile.available())  {
    int n = read(buf, BUF_DIM);
    //  could check for read error with "if (n == 0)"
    client.write(buf, n);
  }

Logged

Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks guys! I have just tried to do single byte read and fill buffer of 90 bytes and send the buffer with one client.print. The speed is much faster (about 20KB over wifi).

I tested just reading one byte at a time and it took 1.6s to read the 43KB file. The SD library is definite not the problem. The client.write is. It probably sends a very short package via wifi every time it is called. After using the client.print() it's all good.

I will do the read multiple byte tonight too!
Logged


Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Just like the usual arduino way, this int read(char*,int) function is in SD.h but never commented or mentioned on their library website:

http://arduino.cc/en/Reference/SD

Such a sad thing! They could have done it better, much better than this. How can arduino be trusted with anything serious?!

Same for client.write(char*,int);

http://arduino.cc/en/Reference/WiFiClientWrite

Just for fairness, here IS some comments:

Code:
// buffered read for more efficient, high speed reading
int File::read(void *buf, uint16_t nbyte) {
  if (_file)
    return _file->read(buf, nbyte);
  return 0;
}

You will need to guess what nbyte could do and what return values could assume in various conditions. Do they even know a library should be like?
« Last Edit: May 04, 2013, 12:45:44 am by liudr » Logged


Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

The file download speed with single byte SD read is about 6-7KB/s when I tried to time a larger file over 100KB in size. After I changed to block read (block size 90 bytes), the upload speed is about 12KB/s up to 16KB/s. But when I use IE, it blocks file download and after that my server is frozen. I'll post this portion of my code first. Anything that is apparently wrong?

This reads the file name from URI and sends the file:

Code:
void csv_file_download(WiFiClient* client, const char filename[])
{
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open(filename, FILE_READ);
  if (myFile)
  {
    client->println("HTTP/1.1 200 OK");
    client->println("Content-Type: text/csv");
    client->println("Connnection: close");
    client->println();
    unsigned long start=millis();
    while(myFile.available())
    {
      int n=myFile.read(buffer,max_wifi_package_length); // multiple byte read.
      if (client->connected()) client->print(buffer); // Only send if the client is still connected.
      else
      {
        Serial.println("Cliented disconnected before file complete.");
        break;
      }
    }
    client->println();
    myFile.close();
    Serial.print(millis()-start);
    Serial.print("ms \r\nSent file:");
    Serial.println(filename);
  }
  else
  {
    client->println("HTTP/1.1 404 Not Found");
    client->println("Content-Type: text/html");
    client->println("Connnection: close");
    client->println();
    client->println("<!DOCTYPE HTML>");
    client->println("<html>HTTP 404 Not Found</html>");
    client->println();
   
    Serial.print("error opening file:");
    Serial.println(filename);
  }
}
Logged


Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Well, worse than I thought. The client.write(uint8_t*,size_t) is not working:

Code:
size_t WiFiClient::write(const uint8_t *buf, size_t size) {
  if (_sock >= MAX_SOCK_NUM)
  {
  setWriteError();
  return 0;
  }
  if (size==0)
  {
  setWriteError();
      return 0;
  }


  if (!ServerDrv::sendData(_sock, buf, size))
  {
  setWriteError();
      return 0;
  }
  if (!ServerDrv::checkDataSent(_sock))
  {
  setWriteError();
      return 0;
  }

  return size;
}

No file was sent to the client and subsequent connections are all failed as arduino stops responding. After I commented out the client.write and uncommended client.print, the download works again.
Logged


Central MN, USA
Offline Offline
Tesla Member
***
Karma: 74
Posts: 7260
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

After adding some more features to my simple file server, such as telling client file size, and checking whether the connection is closed before sending, the server is stable now.
Code:
Code:
    client->println("HTTP/1.1 200 OK");
    client->println("Content-Type: text/csv");
    client->print("Content-Length: ");
    client->println(myFile.size());
    client->println("Connnection: close");
    client->println();
    while(myFile.available())
    {
      int n=myFile.read(buffer,max_wifi_package_length); // multiple byte read.
      if (client->connected())
      {
        client->write((uint8_t*)buffer,n); // Only send if the client is still connected.
      }
      else
      {
        Serial.println("Client disconnected before file complete.");
        break;
      }
    }
    client->println();
    myFile.close();
Logged


Pages: [1]   Go Up
Jump to: