Pages: [1]   Go Down
Author Topic: SdFatLib Question - What should read bandwidth be?  (Read 1752 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 10
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Forgive me if this has been answered elsewhere but I haven't been able to find the answer. 

I've been playing around with Arduino based web servers and I wanted to check to see if the performance I'm seeing reading data from an SD card is in the ball park.

Basically, I have a https://www.sparkfun.com/products/10864 connected to an Uno R3; I'm not sure what brand of SD card I have but it is 2GB formatted FAT16.  When I run the "bench" example from https://code.google.com/p/sdfatlib/downloads/detail?name=sdfatlib20130629.zip&can=2&q=, the maximum read speed is ~280KB/sec.  I'm running in SPI_FULL_SPEED mode so at 8MHz, I'd expect read speeds closer to 1MB/sec.

The problem I am running into is that switching between reading from the SD card via SPI and the sending the data to the Wiz5100, via the same SPI bus, it is taking 5 seconds to download a ~500KB page and I am trying to determine if there is anything I can do to shorten those load times (besides the obvious of making the pages smaller smiley-roll-blue).

Thanks in advance,
Brett.

Edit:  I ran the bench sketch again with the buffer size set to 128 to see the difference.  Here are the results:

Type any character to start

Free RAM: 998

Type is FAT16

File size 5MB

Buffer size 128 bytes

Starting write test.  Please wait up to a minute

Write 170.19 KB/sec

Maximum latency: 278520 usec, Minimum Latency: 100 usec, Avg Latency: 746 usec

 

Starting read test.  Please wait up to a minute

Read 316.65 KB/sec

Maximum latency: 2468 usec, Minimum Latency: 100 usec, Avg Latency: 398 usec

 

Done
« Last Edit: July 17, 2013, 10:55:25 pm by bbbowden » Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You results are typical for small reads that are not a multiple of 512 bytes on an AVR Arduino.

You will not get close to 1 MB/sec for the following reasons.

The reads are done in single block mode so there is no pipe lining in the SD.  Data is read from flash to an internal buffer in the SD then sent over SPI to the arduino.

AVR SPI is not DMA.  There is limited overlap between bus transfer and storing data in memory and a status check is required for every byte.

Small reads require an extra copy from the block buffer to the read buffer.

Here is a typical result for 512 byte reads using an Uno:
Quote
Type is FAT16
File size 5MB
Buffer size 512 bytes
Starting write test.  Please wait up to a minute
Write 493.75 KB/sec
Maximum latency: 59692 usec, Minimum Latency: 948 usec, Avg Latency: 1031 usec

Starting read test.  Please wait up to a minute
Read 549.90 KB/sec
Maximum latency: 1848 usec, Minimum Latency: 916 usec, Avg Latency: 925 usec
The best read speed I have seen on an AVR is about 620 KB/sec.

Here is an SdFat test on a Due for large multi-block reads using DMA:
Quote
Type is FAT16
File size 10MB
Buffer size 16384 bytes
Starting write test.  Please wait up to a minute
Write 4038.08 KB/sec
Maximum latency: 65420 usec, Minimum Latency: 3850 usec, Avg Latency: 4052 usec

Starting read test.  Please wait up to a minute
Read 4406.63 KB/sec
Maximum latency: 3934 usec, Minimum Latency: 3703 usec, Avg Latency: 3715 usec

SD cards are designed to give best performance using large multi-block transfers.  This is how the class 10 and other speed tests are specified.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 10
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Fat16lib,

Thank you, that was exactly the answer I was looking for.  Actually the part about the Due accessing data faster might just be the answer I REALLY needed.  I have one of those too so I'll do some experimenting to see what kind of performance the Due will give my pages.

Thanks again
Brett
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 10
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Fat16lib,

Running into an issue using the SdFat library with the Ethernet library and I was hoping you might be able to help me with it.  The issues I am running into are:
1. When using the SD wrapper library, the code below works but the maximum size of the buffer is 2048 bytes.  If I go larger than that, the data delivered to the browser starts getting divided in half for each power of 2 increase, i.e. 200K file will only be 100K with a 4096 byte buffer, 50K with a 8192 byte buffer, etc.
2. If I try to use SdFat directly, I can't get the Ethernet library to work after initializing the SdFat library.  I'm guess this has to do with the Extended SPI features of the Due, but I have yet to figure out why it won't work and I'm out of ideas.

Any help would be appreciated.

Thanks,
Brett.

Hardware: Arduino Due
               Sparkfun PoE Ethernet Shield (part number above)

Arduino IDE 1.5.2
Latest version of your SdFat Library.

Code:
/*--------------------------------------------------------------
  Program:      eth_websrv_SD_image

  Description:  Arduino web server that serves up a basic web
                page that displays an image.
 
  Hardware:     Arduino Uno and official Arduino Ethernet
                shield. Should work with other Arduinos and
                compatible Ethernet shields.
                2Gb micro SD card formatted FAT16
               
  Software:     Developed using Arduino 1.0.5 software
                Should be compatible with Arduino 1.0 +
               
                Requires index.htm, page2.htm and pic.jpg to be
                on the micro SD card in the Ethernet shield
                micro SD card socket.
 
  References:   - WebServer example by David A. Mellis and
                  modified by Tom Igoe
                - SD card examples by David A. Mellis and
                  Tom Igoe
                - Ethernet library documentation:
                  http://arduino.cc/en/Reference/Ethernet
                - SD Card library documentation:
                  http://arduino.cc/en/Reference/SD

  Date:         7 March 2013
  Modified:     17 June 2013
 
  Author:       W.A. Smith, http://startingelectronics.com
--------------------------------------------------------------*/

#include <SPI.h>
#include <Ethernet.h>
//#include <SD.h>

#include <SdFat.h>
#include <SdFatUtil.h>

// size of buffer used to capture HTTP requests
#define REQ_BUF_SZ   20
#define SDCHUNKS 512

byte sdbuffer[SDCHUNKS];

// MAC address from Ethernet shield sticker under board
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(10, 0, 96, 99); // IP address, may need to change depending on network
EthernetServer server(80);  // create a server at port 80

SdFat sd;
SdFile file;

//File webFile;
char HTTP_req[REQ_BUF_SZ] = {0}; // buffered HTTP request stored as null terminated string
char req_index = 0;              // index into HTTP_req buffer

void setup()
{
    // disable Ethernet chip
    //pinMode(10, OUTPUT);
    //digitalWrite(10, HIGH);
   
    //SPI.begin(4);
    //SPI.begin(10);
   
    Serial.begin(9600);       // for debugging
   
    // initialize SD card
   
    // check for index.htm file
//    if (!SD.exists("index.htm")) {
//        Serial.println("ERROR - Can't find index.htm file!");
//        return;  // can't find index file
//    }

//    if (!file.open("index.htm")) {
//        Serial.println("ERROR - Can't find index.htm file!");
//        return;
//    }
//    Serial.println("SUCCESS - Found index.htm file.");
//    file.close();
   
    Ethernet.begin(mac, ip);  // initialize Ethernet device
   
    Serial.println("Initializing SD card...");
   
    if (!sd.begin(4, 7)) {
      Serial.println("ERROR - SD card initialization failed!");
      return;
    }   
   
//    if (!SD.begin(4)) {
//        Serial.println("ERROR - SD card initialization failed!");
//        return;    // init failed
//    }
    Serial.println("SUCCESS - SD card initialized.");
    server.begin();           // start to listen for clients
}

void loop()
{
    EthernetClient client = server.available();  // try to get client

    if (client) {  // got client?
        boolean currentLineIsBlank = true;
        while (client.connected()) {
            if (client.available()) {   // client data available to read
                char c = client.read(); // read 1 byte (character) from client
                // buffer first part of HTTP request in HTTP_req array (string)
                // leave last element in array as 0 to null terminate string (REQ_BUF_SZ - 1)
                if (req_index < (REQ_BUF_SZ - 1)) {
                    HTTP_req[req_index] = c;          // save HTTP request character
                    req_index++;
                }
                // print HTTP request character to serial monitor
                Serial.print(c);
                // last line of client request is blank and ends with \n
                // respond to client only after last line received
                if (c == '\n' && currentLineIsBlank) {
                    // open requested web page file
                    if (StrContains(HTTP_req, "GET / ")
                                 || StrContains(HTTP_req, "GET /index.htm")) {
                        client.println("HTTP/1.1 200 OK");
                        client.println("Content-Type: text/html");
                        client.println("Connnection: close");
                        client.println();
                        //webFile = SD.open("index.htm");        // open web page file
                        file.open("index.htm");
                    }
                    else if (StrContains(HTTP_req, "GET /page2.htm")) {
                        client.println("HTTP/1.1 200 OK");
                        client.println("Content-Type: text/html");
                        client.println("Connnection: close");
                        client.println();
                        //webFile = SD.open("page2.htm");        // open web page file
                        file.open("page2.htm");
                    }
                    else if (StrContains(HTTP_req, "GET /pic.jpg")) {
                        //webFile = SD.open("pic.jpg");
                       
                        if (file.open("pic.jpg")) {
                            client.println("HTTP/1.1 200 OK");
                            client.println();
                        }
                    }
                    else if (StrContains(HTTP_req, "GET /bigpic.jpg")) {
                        //webFile = SD.open("bigpic.jpg");
                       
                        if (file.open("bigpic.jpg")) {
                            client.println("HTTP/1.1 200 OK");
                            client.println();
                        }
                    }
                    //if (webFile) {
                        while(file.available()) {
                            int c = file.read((char*)sdbuffer, SDCHUNKS);
                            client.write(sdbuffer, c); // send web page to client
                        }
                        file.close();
                    //}
                    // reset buffer index and all buffer elements to 0
                    req_index = 0;
                    StrClear(HTTP_req, REQ_BUF_SZ);
                    break;
                }
                // every line of text received from the client ends with \r\n
                if (c == '\n') {
                    // last character on line of received text
                    // starting new line with next character read
                    currentLineIsBlank = true;
                }
                else if (c != '\r') {
                    // a text character was received from client
                    currentLineIsBlank = false;
                }
            } // end if (client.available())
        } // end while (client.connected())
        delay(1);      // give the web browser time to receive the data
        client.stop(); // close the connection
    } // end if (client)
}

// sets every element of str to 0 (clears array)
void StrClear(char *str, char length)
{
    for (int i = 0; i < length; i++) {
        str[i] = 0;
    }
}

// searches for the string sfind in the string str
// returns 1 if string found
// returns 0 if string not found
char StrContains(char *str, char *sfind)
{
    char found = 0;
    char index = 0;
    char len;

    len = strlen(str);
   
    if (strlen(sfind) > len) {
        return 0;
    }
    while (index < len) {
        if (str[index] == sfind[found]) {
            found++;
            if (strlen(sfind) == found) {
                return 1;
            }
        }
        else {
            found = 0;
        }
        index++;
    }

    return 0;
}
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 10
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Fat16Lib,

I was reading your documentation (I know, do that first BEFORE asking questions) and tried the
Quote
You can use the standard SPI.h library by editing SdFatConfig.h and set
USE_ARDUINO_SPI_LIBRARY nonzero.  You must include SPI.h in your sketch.
line.  When I tried it, I ran into a typo on line 52 in SdSpiArduino.cpp:
Code:
uint8_t SdSpi::receive(uint8_t* buf, size_t n) {
  for (size_t i = 0; i < n; i++) {
    uf[i] = SPI.transfer(0XFF);
  }
  return 0;
}

I believe it should read:
Code:
uint8_t SdSpi::receive(uint8_t* buf, size_t n) {
  for (size_t i = 0; i < n; i++) {
    buf[i] = SPI.transfer(0XFF);
  }
  return 0;
}
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 10
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Nevermind, I figured out the answers to my two questions:

1. The error of less than 2048 bytes has to do with the limitation of the Ethernet library.  The W5100 library has a 2048 byte buffer that limits the amount of data that can be sent to the W5100 at one time.  I guess I'll have to break down the data into 2048 byte packets myself.
2. Right in the SdFat documentation, it suggests using the standard Arduino SPI library.  That resolved the shared SPI bus issue and allowed me to find the answer to #1.
« Last Edit: July 30, 2013, 06:04:47 pm by bbbowden » Logged

Pages: [1]   Go Up
Jump to: