I hope it is okay that this is not quite a typical "here's my question" post. I hope it will be helpful in a practical way, but also have some unexplained things at the end in case you want to solve a mystery.
Short version: Use a read buffer of 128 to 4096 bytes and call
SPI.setClockDivider(SPI.getClockDivider()*2);
before using the SD card. For more on how I got there and speed vs. buffer size, keep reading.
EDIT: The clock divider does not work the way I thought. There seems to be a max of 4194303 (0x3FFFFF). Currently (board package 2.0.13) the get call gives 2363393. If I multiply by two, set that value, and immediately do another getClockDivider() the value has been wrapped to a smaller value (532482 in this case). I'm not sure my little trick is even relevant now, but if you want to try a larger divider use 2363393 < new value <= 4194303.
In fact, my latest test suggest not messing with this at all. Some reasonable-looking values give unreliable results, and the default is working well. Copying a file of just over 1MB to a new name takes 6.7 to 6.9 seconds.
I have a project where we log data to a microSD card. On older Arduinos we just pulled the card after each experiment and copied the file to a laptop. With the ESP32 it's now easy (?) to just download it.
The file will normally be perhaps 1 MB in size, but someone let one grow to 77 MB so I used it for testing and got some very strange behavior. It turns out that there are 4 ways to get data off the card.
- Byte by byte. Very slow. Forget it.
- Use a buffer and the library defaults. Works on many reads, but fails randomly.
- Use buffer and defaults, catching errors and retrying reads. Reliable, but with inconsistent speed.
- Lower the SPI clock rate. Hey it just works! Speed vs. buffer size has some surprises.
Approaches 2 and 4 use the same code to send the data. Of course the file and connection have be set up and closed before and after.
char buf[bSize];
while (file.available()) {
rlen = file.readBytes(buf, bSize);
client.write(buf, rlen);
}
In the original (approach 2) my computer would report a decent (say 2.4 Mbps) bit rate for a while and then the data would stop. The loop would still be running because available() was returning true even though readBytes() was returning 0! Because it worked for a while and stopped I suspected that I was overflowing some buffer. Adding some delay() calls after every 100 reads seemed to make things better, but it was still failing in the end.
Approach 3 involved a lot of ugly code that I don't recommend using. In very rough pseudocode, the inside of the while above becomes:
rlen = file.readBytes(buf, bSize);
while (rlen == 0 and less than 10 retries) {
// Sometimes we just need to re-read, but other times the file pointer
// has jumped to around 4 GB.
if (file.position() beyond the known file size) {
close the file
while (!SD.begin()) {
// This eventually works, but can take 20 or more passes.
delay(500);
}
open the file again
}
seek(last good position)
retry the readBytes
}
if rlen is still zero after the retries, give up
save the current position as the last good one
client.write(buf, rlen);
continue outer while
The code caught 3 kinds of errors and got around them, but it left the errors unexplained and performance was all over the map, say 75 to 270 KB per second.
After reading many articles on many web sites that didn't help I managed to do my own thinking. What's different about the ESP32 that would matter? It's fast. I tried changing the SPI clock divider and that was the magic bullet. With
SPI.setClockDivider(SPI.getClockDivider()*2);
the errors all went away. By the way, the default divisor was 4097.
That got me a steady speed of about 290 KB per second. A reasonable person would stop there and move on with the project. I decided to switch to a smaller 1 MB file and play with the buffer size.
The first thing is pretty obvious. Very small buffers are slower. A 64 byte buffer requires almost 11 seconds to move a file which takes under 4 seconds with a better choice. It's the sort of smoothly dropping curve you might expect. I didn't show it here, but speeds actually get a little worse with a buffer above 5000 bytes.
Less obvious is that there is a very hard limit out there. I didn't nail it down to the byte, but a 6000 byte buffer or less works every time. 7000 or more causes a reboot every time on the first read. I would have expected a power of 2. Anyone understand this?
Even less obvious to me is that speeds suddenly get better going from 4306 to 4308 bytes. Even stranger, 4307 is terrible. There's no big change at 4096, but a 1 MB transfer
that takes about 3500 ms with a buffer size up to 4306 bytes takes about 5800 ms with a 4307 byte buffer and only about 2900 with a 4308 byte buffer!
I ran that 4307 point 3 times and it was always 5744 or higher. None of the nearby points was higher than 3730 ms. What's going on?
Probably the practical thing is to stop with a 4096-byte buffer, but the desire to optimize is strong.
Oh - for what it's worth the timings shown were done with a new SanDisk Ultra 16 GB card, HC, class 10. Nothing fancy, but decent. I got similar bug/non bug behavior with an older 8 GB HC class 4 card. Speeds were essentially the same in a quick test and the reboot happened at 6656 bytes but not at 6144 or below, the same as with the newer card.
The Nano ESP32 is on a custom circuit board with an SD slot surface mounted to it. It uses the default SPI pins with card select on D5. There is a 1 uF capacitor between power and ground close to the slot. Voltage measures at 3.24 volts. A TFT display which uses SPI was connected, but just displaying a constant text string so it should never have been active during the tests. There's not much else going on on the PCB, just a real time clock, a shift register, and a few connectors.