I heard a rumour about a way to optimize the transferring of files from SD-card to Ethernet. Now I use this code:
while (dataFile.available()) {
c.write(dataFile.read());
wdt_reset(); // reset watchdog timer
}
dataFile.close();
…and I upload about 37000 bits per second.
Anyone with a better way?
It's hard to tell from you snippet but are you really sending one byte at a time? Wouldn't it be faster to assemble them into packet-size chunks?
OP,
Was that really 37000 BITs? That is not very impressive, like 4.6KB per second, kind of depressing for me who's thinking about doing it. Have you tried Nick's suggestion to read and write in large buffer size instead of one byte at a time?
Server test code that serves the HYPNO.JPG pix from the SD card.
info link
http://forum.arduino.cc/index.php/topic,134868.0.html
//zoomkat 12/26/12
//SD server test code
//open serial monitor to see what the arduino receives
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields
#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>
byte mac[] = {
0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = {
192, 168, 1, 102 }; // ip in lan
byte gateway[] = {
192, 168, 1, 1 }; // internet access via router
byte subnet[] = {
255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
String readString;
//////////////////////
void setup(){
Serial.begin(9600);
// disable w5100 while setting up SD
pinMode(10,OUTPUT);
digitalWrite(10,HIGH);
Serial.print("Starting SD..");
if(!SD.begin(4)) Serial.println("failed");
else Serial.println("ok");
Ethernet.begin(mac, ip, gateway, gateway, subnet);
//delay(2000);
server.begin();
Serial.println("Ready");
}
void loop(){
// Create a client connection
EthernetClient client = server.available();
if (client) {
while (client.connected()) {
if (client.available()) {
char c = client.read();
//read char by char HTTP request
if (readString.length() < 100) {
//store characters to string
readString += c;
//Serial.print(c);
}
//if HTTP request has ended
if (c == '\n') {
///////////////
Serial.println(readString); //print to serial monitor for debuging
client.println("HTTP/1.1 200 OK"); //send new page
//client.println("Content-Type: text/html");
client.println("Content-Type: image/jpeg");
//client.println("Content-Type: image/gif");
//client.println("Content-Type: application/x-javascript");
//client.println("Content-Type: text");
client.println();
//File myFile = SD.open("boom.htm");
File myFile = SD.open("HYPNO.JPG");
//File myFile = SD.open("BLUEH_SL.GIF");
//File myFile = SD.open("SERVOSLD.HTM");
if (myFile) {
byte clientBuf[64];
int clientCount = 0;
while(myFile.available())
{
clientBuf[clientCount] = myFile.read();
clientCount++;
if(clientCount > 63)
{
// Serial.println("Packet");
client.write(clientBuf,64);
clientCount = 0;
}
}
//final <64 byte cleanup packet
if(clientCount > 0) client.write(clientBuf,clientCount);
// close the file:
myFile.close();
}
delay(1);
//stopping client
client.stop();
readString="";
}
}
}
}
}
//store characters to string
readString += c;
Please note that in versions of the IDE up to and including 1.0.3, the String library has bugs as discussed here and here.
In particular, the dynamic memory allocation used by the String class may fail and cause random crashes.
I recommend reworking your code to manage without String. Use C-style strings instead (strcpy, strcat, strcmp, etc.), as described here for example.
Alternatively, install the fix described here: Fixing String Crashes
Preferably upgrade your IDE to version 1.0.4 or above at: http://arduino.cc/en/Main/Software
The time taken to concatenate in the String class is quite substantial, so this isn’t a good idea if you are trying to save time.
Example:
#include <ProfileTimer.h>
void setup ()
{
Serial.begin (115200);
Serial.println ();
String s;
char a [513];
{
ProfileTimer t1 ("concatenating String");
for (int i = 0; i < 512; i++)
s += 'a';
} // end timed bit of code
Serial.println ();
{
ProfileTimer t1 ("concatenating char array");
for (int i = 0; i < 512; i++)
a [i] = 'a';
a [512] = 0; // terminating null
} // end timed bit of code
} // end of setup
void loop () { }
Output:
Start : concatenating String
Time taken: concatenating String = 12240 uS.
Start : concatenating char array
Time taken: concatenating char array = 8 uS.
That String concatenation requires a malloc and free for every byte. That takes time as well as being likely to fragment memory, a scarce resource on a microcontroller.
Using String concatenation took 1530 times as long!
I suggest you read into a fixed “char” array buffer, and then transmit (send) when you have a reasonable number, like 512 bytes.
I think the compiler may have optimized away the char concatenation. The figure of 8 µS is too good to be true. A modified sketch which prints the results (forcing the compiler to calculate them) gives:
Start : concatenating String
Time taken: concatenating String = 12240 uS.
Start : concatenating char array
Time taken: concatenating char array = 212 uS.
It is still a lot slower using String, but only 58 times as slow. 
This is the important part of the transfer from zoomkat’s code. This sends 64 byte payloads. About 4 times faster than sending one byte at a time.
byte clientBuf[64];
int clientCount = 0;
while(myFile.available())
{
clientBuf[clientCount] = myFile.read();
clientCount++;
if(clientCount > 63)
{
// Serial.println("Packet");
client.write(clientBuf,64);
clientCount = 0;
}
}
//final <64 byte cleanup packet
if(clientCount > 0) client.write(clientBuf,clientCount);
// close the file:
myFile.close();
Thanks for you replies!
I put in zoomkat's changes and the result now looks like this:
#define BUFSIZE 32
byte clientBuf[BUFSIZE];
int clientCount = 0;
while (dataFile.available()) {
clientBuf[clientCount] = dataFile.read();
clientCount++;
if(clientCount > BUFSIZE-1)
{
// Serial.println("Packet");
c.write(clientBuf,BUFSIZE);
clientCount = 0;
}
wdt_reset(); // reset watchdog timer
}
if(clientCount > 0) c.write(clientBuf,clientCount); // Skicka de sista byten
The reward was a much higher transfer rate. My tests, counting seconds of transfer time for a 591000 byte picture sent to Firefox via LAN, showed a nice improvement.
105000 bit/s at 8 byte buffer
124000 bit/s at 16 byte buffer
139000 bit/s at 32 byte buffer
143000 bit/s at 64 byte buffer
148000 bit/s at 128 byte buffer
I connected a logic analyser to the SPI-bus and I could see that 2.7ms of the time (9%) is used to communicate with the SD-card (hard to count, but 500B to 1kB is transferred), and during the rest of the time the bus speaks to the W5100 (about 180 bytes transferred), waits 1.17ms and repeats 15 more time until it is finished after 27.3ms. Then this sequence is repeated until the file is sent. SPI CLOCK, MISO and MOSI runs at 4MHz on the ICSP-connector, not on the ordinary pins.
So I guess that it is the communication with the W5100 that can be improved.
Yep,
14KB/s is what I was expecting.