Using SdFat with ESPAsyncWebSrv

Official Hardware Nano Family Nano ESP32
1

First, why this combination?

  • The web server is fast and easy to use, and doesn't slow down other processes noticeably. I have already written a lot of code using it.
  • SD.h is buggy. It randomly decides that I have no SD card, any time from just after boot to after a few hundred thousand reads and writes. Repeating SD.begin doesn't help, and often a reboot doesn't help either.
  • SdFat is working first time, every time on the Nano ESP32.

When I was happy with microSD tests using SdFat I converted my main project which includes the web server. Everything went sideways. ESP32AsyncWebSrv includes FS.h and uses File objects. These conflict with similar definitions in SdFat.h. There are also conflicting #defines for things like FILE_WRITE, though I'm happy to use the primitives and skip the shorthand names.

Just one example of the problem is
server.serveStatic("/", SD, "/htdocs").setCacheControl("max-age=3600");
where I have SD defined based on SdFat.h but the server code expects the FS from FS.h.

The compiler output for this (amid many other warnings and errors) is

D:\GitHub\CBASS-ESP32-Sketch\CBASS_WiFi_Async\Server.ino: In function 'void defineWebCallbacks()':
D:\GitHub\CBASS-ESP32-Sketch\CBASS_WiFi_Async\Server.ino:64:40: error: no matching function for call to 'AsyncWebServer::serveStatic(const char [2], SdFat32&, const char [8])'
   server.serveStatic("/", SD, "/htdocs").setCacheControl("max-age=3600");  // for a full day: 86400");
                                        ^
In file included from D:\GitHub\CBASS-ESP32-Sketch\CBASS_WiFi_Async\CBASS_WiFi_Async.ino:61:
C:\Users\list\Documents\Arduino\libraries\ESPAsyncWebSrv\src/ESPAsyncWebSrv.h:428:28: note: candidate: 'AsyncStaticWebHandler& AsyncWebServer::serveStatic(const char*, fs::FS&, const char*, const char*)'
     AsyncStaticWebHandler& serveStatic(const char* uri, fs::FS& fs, const char* path, const char* cache_control = NULL);
                            ^~~~~~~~~~~
C:\Users\list\Documents\Arduino\libraries\ESPAsyncWebSrv\src/ESPAsyncWebSrv.h:428:28: note:   no known conversion for argument 2 from 'SdFat32' to 'fs::FS&'

Obviously this can't work, but I'm at a loss for how to proceed from here. I could try to edit the web server to use SdFat. I'm okay with modifying hundreds of lines of code, but it feels like asking for trouble as new versions come along.

Some people seem to have had success with explicit namespaces (SD.h uses fs, but I haven't spotted such a name in SdFat). One thing I do NOT want to do is use both SdFat.h and SD.h. Given recent experiences, the built-in SD.h is simply dangerous.

There are other forum posts with similar issues, but they seem to be 2 to 12 years old. I'm hoping that there is some new knowledge out there that I can build on.

What do you suggest I do next?

Some details:

  • Arduino IDE 2.3.2
  • Board definition 2.0.13
  • SdFat by Bill Greiman, 2.2.2
  • ESPAsyncWebSrv by dvarrle, 1.2.7
  • There are two SPI devices. The other is a TFT display. The display was included in my successful SdFat tests.
  • Running on Arduino Nano ESP32
  • Compiling on Windows 11 Pro
  • The problem SD.h I'm trying to avoid is this one: C:\Users\xxx\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.13\libraries\SD\src\SD.h
  • microSD read/write speed is not important. I'm only logging a hundred bytes or so once per second, and reading only a few small files per run. Web use will be light.
  • I can post all 4,000 lines of code or put it on GitHub, but I'm hoping that an general response will get me going. I'll post my results.

Things I tried with SD.h before going to SdFat:

  1. More than one copy of my custom PCB where the sdCard slot lives.
  2. Different brands and sizes of microSD card.
  3. Rebooting with the reset button or by physically removing power. Times of at least 20 seconds seemed better, but not consistently.
  4. Voltage on the Nano consistently tests at 3.24 or 3.25 with or without a card installed. This is with a basic hobbyist-level digital meter, and could be off a bit.
  5. Adding extra pullup resistors.
  6. Catching and retrying failed SD operations.
  7. Changing the SPI clock divider.

Those last few items had no effect or made things worse, so they are all back at defaults.

Thanks for anything!

2

I have some examples in my webserver libraries (basically it's the same example but one based on ESPAsyncWebServer (the original from me-no-dev) and the other on the WebServer library included in the ESP32/ESP8266 Arduino core) and I have never encountered issues with reliability of SD or anything else.

I have to say I'm not using Arduino Nano ESP32, but I think is the same as "no-brand" ESP32 or not?

Note:
I would suggest staying on the core library because it's maintained, while ESPAsyncWebServer, although excellent, is no longer followed by the "real" developer (the one you installed is just a copy that follow the Arduino library registry requirements)

3

Thanks for the reply. It sounds like I should switch web servers, but it doesn't solve my SD.h reliability problem. I should have made clear in the first post that the reliability issue appears with or without the web server.

I have a version where I stripped out the web server, sensor readings, setting of relays, and other functions until there was just the SD, a real time clock, and the TFT display (sold by Adafruit).
It takes me down to just these includes

#include <esp_task_wdt.h>
#include <Adafruit_ILI9341.h>
#include <SD.h>
#include <RTClib.h>

SD is from Arduino15\packages\arduino\hardware\esp32\2.0.13\libraries\SD, so it should be the core version.

It is this code where I fairly easily switched to SdFat and the problems went away.

Last night I built a version of ESPAsyncWebServ using SdFat and tried it with my full version. It serves files from the microSD correctly, but it's not a drop-in replacement and I have created some bugs. Of course if I switch servers I can just delete that version.

The Nano uses the ESP32-S3 in a NORA-W106 module. I don't know whether that makes a difference compared to your system. It has been very reliable in every other way, and I have been testing on more than one copy.

New this morning: another thing I did yesterday was follow some old online advice that says to use a downloadable formatting tool instead of the one in my OS. For what it's worth I haven't seen the reliability problem today.

Most of the code (but not the web parts) is from a version which runs on Arduino Megas and clones with the ATmega2560 chip. Dozens of devices have run for many thousands of hours and I have never heard of a data access issue. The microSD cards were presumably formatted on Mac, PC, or most often used straight from the retail package. Maybe the formatting is more critical at the higher speeds of the ESP32.

4

No, I'm just saying that I never experienced this issues with web servers and standard SD, off course SD problems don't depend from server.

Have you considered serving your content from SD without using serveStatic() method?

You could read the file using the SdFat API into a buffer and send the buffer instead of passing the filesystem reference and file name.

5

Ooh - I like the idea of avoiding serveStatic(). I'll have to try that. Most things are in PROGMEM or dynamically generated anyway. I can certainly serve the handful of static files as you describe.

Thanks.