Forgive me if this has been answered elsewhere but I haven't been able to find the answer.
I've been playing around with Arduino based web servers and I wanted to check to see if the performance I'm seeing reading data from an SD card is in the ball park.
The problem I am running into is that switching between reading from the SD card via SPI and the sending the data to the Wiz5100, via the same SPI bus, it is taking 5 seconds to download a ~500KB page and I am trying to determine if there is anything I can do to shorten those load times (besides the obvious of making the pages smaller ).
Thanks in advance,
Brett.
Edit: I ran the bench sketch again with the buffer size set to 128 to see the difference. Here are the results:
You results are typical for small reads that are not a multiple of 512 bytes on an AVR Arduino.
You will not get close to 1 MB/sec for the following reasons.
The reads are done in single block mode so there is no pipe lining in the SD. Data is read from flash to an internal buffer in the SD then sent over SPI to the arduino.
AVR SPI is not DMA. There is limited overlap between bus transfer and storing data in memory and a status check is required for every byte.
Small reads require an extra copy from the block buffer to the read buffer.
Here is a typical result for 512 byte reads using an Uno:
Type is FAT16
File size 5MB
Buffer size 512 bytes
Starting write test. Please wait up to a minute
Write 493.75 KB/sec
Maximum latency: 59692 usec, Minimum Latency: 948 usec, Avg Latency: 1031 usec
Starting read test. Please wait up to a minute
Read 549.90 KB/sec
Maximum latency: 1848 usec, Minimum Latency: 916 usec, Avg Latency: 925 usec
The best read speed I have seen on an AVR is about 620 KB/sec.
Here is an SdFat test on a Due for large multi-block reads using DMA:
Type is FAT16
File size 10MB
Buffer size 16384 bytes
Starting write test. Please wait up to a minute
Write 4038.08 KB/sec
Maximum latency: 65420 usec, Minimum Latency: 3850 usec, Avg Latency: 4052 usec
Starting read test. Please wait up to a minute
Read 4406.63 KB/sec
Maximum latency: 3934 usec, Minimum Latency: 3703 usec, Avg Latency: 3715 usec
SD cards are designed to give best performance using large multi-block transfers. This is how the class 10 and other speed tests are specified.
Thank you, that was exactly the answer I was looking for. Actually the part about the Due accessing data faster might just be the answer I REALLY needed. I have one of those too so I'll do some experimenting to see what kind of performance the Due will give my pages.
Running into an issue using the SdFat library with the Ethernet library and I was hoping you might be able to help me with it. The issues I am running into are:
When using the SD wrapper library, the code below works but the maximum size of the buffer is 2048 bytes. If I go larger than that, the data delivered to the browser starts getting divided in half for each power of 2 increase, i.e. 200K file will only be 100K with a 4096 byte buffer, 50K with a 8192 byte buffer, etc.
If I try to use SdFat directly, I can't get the Ethernet library to work after initializing the SdFat library. I'm guess this has to do with the Extended SPI features of the Due, but I have yet to figure out why it won't work and I'm out of ideas.
Any help would be appreciated.
Thanks,
Brett.
Hardware: Arduino Due
Sparkfun PoE Ethernet Shield (part number above)
Arduino IDE 1.5.2
Latest version of your SdFat Library.
/*--------------------------------------------------------------
Program: eth_websrv_SD_image
Description: Arduino web server that serves up a basic web
page that displays an image.
Hardware: Arduino Uno and official Arduino Ethernet
shield. Should work with other Arduinos and
compatible Ethernet shields.
2Gb micro SD card formatted FAT16
Software: Developed using Arduino 1.0.5 software
Should be compatible with Arduino 1.0 +
Requires index.htm, page2.htm and pic.jpg to be
on the micro SD card in the Ethernet shield
micro SD card socket.
References: - WebServer example by David A. Mellis and
modified by Tom Igoe
- SD card examples by David A. Mellis and
Tom Igoe
- Ethernet library documentation:
http://arduino.cc/en/Reference/Ethernet
- SD Card library documentation:
http://arduino.cc/en/Reference/SD
Date: 7 March 2013
Modified: 17 June 2013
Author: W.A. Smith, http://startingelectronics.com
--------------------------------------------------------------*/
#include <SPI.h>
#include <Ethernet.h>
//#include <SD.h>
#include <SdFat.h>
#include <SdFatUtil.h>
// size of buffer used to capture HTTP requests
#define REQ_BUF_SZ 20
#define SDCHUNKS 512
byte sdbuffer[SDCHUNKS];
// MAC address from Ethernet shield sticker under board
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(10, 0, 96, 99); // IP address, may need to change depending on network
EthernetServer server(80); // create a server at port 80
SdFat sd;
SdFile file;
//File webFile;
char HTTP_req[REQ_BUF_SZ] = {0}; // buffered HTTP request stored as null terminated string
char req_index = 0; // index into HTTP_req buffer
void setup()
{
// disable Ethernet chip
//pinMode(10, OUTPUT);
//digitalWrite(10, HIGH);
//SPI.begin(4);
//SPI.begin(10);
Serial.begin(9600); // for debugging
// initialize SD card
// check for index.htm file
// if (!SD.exists("index.htm")) {
// Serial.println("ERROR - Can't find index.htm file!");
// return; // can't find index file
// }
// if (!file.open("index.htm")) {
// Serial.println("ERROR - Can't find index.htm file!");
// return;
// }
// Serial.println("SUCCESS - Found index.htm file.");
// file.close();
Ethernet.begin(mac, ip); // initialize Ethernet device
Serial.println("Initializing SD card...");
if (!sd.begin(4, 7)) {
Serial.println("ERROR - SD card initialization failed!");
return;
}
// if (!SD.begin(4)) {
// Serial.println("ERROR - SD card initialization failed!");
// return; // init failed
// }
Serial.println("SUCCESS - SD card initialized.");
server.begin(); // start to listen for clients
}
void loop()
{
EthernetClient client = server.available(); // try to get client
if (client) { // got client?
boolean currentLineIsBlank = true;
while (client.connected()) {
if (client.available()) { // client data available to read
char c = client.read(); // read 1 byte (character) from client
// buffer first part of HTTP request in HTTP_req array (string)
// leave last element in array as 0 to null terminate string (REQ_BUF_SZ - 1)
if (req_index < (REQ_BUF_SZ - 1)) {
HTTP_req[req_index] = c; // save HTTP request character
req_index++;
}
// print HTTP request character to serial monitor
Serial.print(c);
// last line of client request is blank and ends with \n
// respond to client only after last line received
if (c == '\n' && currentLineIsBlank) {
// open requested web page file
if (StrContains(HTTP_req, "GET / ")
|| StrContains(HTTP_req, "GET /index.htm")) {
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: text/html");
client.println("Connnection: close");
client.println();
//webFile = SD.open("index.htm"); // open web page file
file.open("index.htm");
}
else if (StrContains(HTTP_req, "GET /page2.htm")) {
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: text/html");
client.println("Connnection: close");
client.println();
//webFile = SD.open("page2.htm"); // open web page file
file.open("page2.htm");
}
else if (StrContains(HTTP_req, "GET /pic.jpg")) {
//webFile = SD.open("pic.jpg");
if (file.open("pic.jpg")) {
client.println("HTTP/1.1 200 OK");
client.println();
}
}
else if (StrContains(HTTP_req, "GET /bigpic.jpg")) {
//webFile = SD.open("bigpic.jpg");
if (file.open("bigpic.jpg")) {
client.println("HTTP/1.1 200 OK");
client.println();
}
}
//if (webFile) {
while(file.available()) {
int c = file.read((char*)sdbuffer, SDCHUNKS);
client.write(sdbuffer, c); // send web page to client
}
file.close();
//}
// reset buffer index and all buffer elements to 0
req_index = 0;
StrClear(HTTP_req, REQ_BUF_SZ);
break;
}
// every line of text received from the client ends with \r\n
if (c == '\n') {
// last character on line of received text
// starting new line with next character read
currentLineIsBlank = true;
}
else if (c != '\r') {
// a text character was received from client
currentLineIsBlank = false;
}
} // end if (client.available())
} // end while (client.connected())
delay(1); // give the web browser time to receive the data
client.stop(); // close the connection
} // end if (client)
}
// sets every element of str to 0 (clears array)
void StrClear(char *str, char length)
{
for (int i = 0; i < length; i++) {
str[i] = 0;
}
}
// searches for the string sfind in the string str
// returns 1 if string found
// returns 0 if string not found
char StrContains(char *str, char *sfind)
{
char found = 0;
char index = 0;
char len;
len = strlen(str);
if (strlen(sfind) > len) {
return 0;
}
while (index < len) {
if (str[index] == sfind[found]) {
found++;
if (strlen(sfind) == found) {
return 1;
}
}
else {
found = 0;
}
index++;
}
return 0;
}
Nevermind, I figured out the answers to my two questions:
The error of less than 2048 bytes has to do with the limitation of the Ethernet library. The W5100 library has a 2048 byte buffer that limits the amount of data that can be sent to the W5100 at one time. I guess I'll have to break down the data into 2048 byte packets myself.
Right in the SdFat documentation, it suggests using the standard Arduino SPI library. That resolved the shared SPI bus issue and allowed me to find the answer to #1.
).