ESP32 Data Logger Occasionally Misses SD Card Writes After Several Hours of Operation

Hi everyone,

I'm working on an ESP32-based environmental monitoring system that collects temperature, humidity, and air quality data and stores the readings on a microSD card every few minutes.

The system works well initially, but after running continuously for several hours (sometimes overnight), I occasionally notice missing log entries in the SD card file. The ESP32 itself doesn't appear to crash or reboot, but some measurements simply never get written.

Current setup:

  • ESP32-WROOM module

  • SPI microSD card module

  • BME280 sensor

  • 5V external power supply

  • Custom PCB

Things I've already checked:

  • SD card is formatted correctly

  • Different SD cards show similar behavior

  • Added delays after write operations

  • Confirmed there are no obvious resets in the serial logs

A few questions:

  1. Have you experienced intermittent SD card write failures on long-running ESP32 projects?

  2. Are there known SPI timing or buffering issues that could cause occasional missed writes?

  3. Would adding local RAM buffering before committing data to the SD card improve reliability?

  4. Are there any PCB layout practices around SPI traces, grounding, or power filtering that could help prevent these types of issues?

I'm preparing the next PCB revision and considering having it manufactured, so I'd appreciate any design recommendations before sending the files out.

Thanks

A RAM buffer is definitely worth trying. Writing sensor data to memory first and committing it to the SD card in larger batches can reduce SPI transactions and improve long-term reliability.

I'd also check the SPI signal quality and power stability during write operations. On custom PCBs, long SPI traces, inadequate decoupling, or noise from the power supply can occasionally cause corrupted or missed transfers without triggering a reset.

For the next revision, keep the SPI traces short, add local decoupling near the SD card module, and maintain a solid ground plane. If you're ordering a new board, PCB manufacturers such as PCBWay can also perform design-rule checks that may help catch layout issues before fabrication.

Good luck with the next revision!

I hadn't really considered that the SD card writes themselves might be the issue rather than the ESP32 application logic.

I'm currently writing each measurement directly to the card as it's collected, so implementing a RAM buffer and batching writes sounds like a good next step.

For those who have deployed long-running data loggers, have you found that SD card reliability is usually limited by software handling, or do PCB layout and power integrity end up being equally important in practice?

I moved your topic to a more appropriate forum category @maria1719.

The Nano Family > Nano ESP32 category you chose is only used for discussions directly related to the Arduino Nano ESP32 board.

In the future, when creating a topic please take the time to pick the forum category that best suits the subject of your question. There is an "About the _____ category" topic at the top of each category that explains its purpose.

Thanks in advance for your cooperation.

No i haven't. That said i haven't used an SD-card for something like data-logging at all, but rather for storing huge chunks (frames) of data for some period of time (anywhere from a few seconds to several hours) and storage speed has been my limitation.

Probably not. I did add a Queue and with that i managed to resolve the speed issue i had, where the SD-writes were holding the processing thread.

The SD library does include a 512 byte buffer, for the time the file is open. Without seeing how you implement the whole thing it is hard to give you advice. Do you close the file at the end of each operation, because doing so will improve reliability over a file that remains open.

Not to my knowledge.

There are a lot of layout practices that one should observe. The obvious is making the SPI traces as short as possible and providing ample current. There are a lot of things that are probably not the best, but in my experience, even running a few Dupont header wires of 10 cm between the SD-card and the ESP32, does not actually impact reliability or speed at all (i still can communicate at 32Mhz with the card), so in short i somehow doubt this is the issue, but without the used schematic and the PCB layout it is really just a guess.

My bet is on a software bug, within your code. I simple thing would actually be to verify the write has gone the way it should by reading the data back from the SD-card after writing.

Thanks for doing so. I'll take care next time.

Thanks for the detailed explanation.
You may be right that I'm focusing too much on the hardware side. Currently, the file stays open for extended periods because I wanted to minimize write overhead, so I haven't been closing it after each logging operation. I'll experiment with more frequent open/write/close cycles and see if the missing entries disappear.

I also like the idea of verifying writes by reading the data back periodically. That should help determine whether the issue is occurring during the write process itself or somewhere else in the application.

At this point, I'm starting to suspect a software issue as well, so I'll spend some time reviewing the logging code before assuming the PCB is at fault.

Thanks again for the suggestions.

Unless you are writing at high speed, there is no need to keep it open. If there is nothing to write, close it. I am confident this is the issue.

I would attach on OpenLog to the serial port pins on the ESP32.

Openlogs are only about £5 and any issues or reported errors with the SD will be written to a log file on the openlog.

You might also want to test your software on an ESP32S3 Camera Development board, that has a built in SD card holder (MMC mode) so all the traces etc ought to be good.

In general I have not found the SD card functions on the ESP32 to be completely reliable.

From my experience, I've had no issues saving data to an SD card using an ESP32, mostly deep sleeping, with a Lipo battery. Over 3 years, around 7 months at a time (each winter), two writes a day to a micro SD card, of approx. 500bytes each write.

Temperature data gathered every 15mins and stored in RTC SRAM. Twice a day, power on the SD card, waited 50ms before SD.begin and SD.open for writing. (Using SD.h library). Write 500bytes of data. Finally, used file 'flush()', then file 'close()'. Turn SD card power off.

The SD card module I used has an AMS1117 regulator on board for 3.3V. I shorted that out and fed the module with 3.3v from the ESP32 board. (Switched to save power during deep sleep). The SPI connections are 2 to 3 inches of loosely twisted wires. Nothing special.
Haven't had any write failures or missing data. They seem pretty robust to me.

I buy 200 sdcard with my logo ecc a good price , but when arrived i cant write with mks-dlc32 esp32 based. when write with pc is ok , both esp32 and pc read, but when try to write with esp32 something happens , the sdcard seem that is wrote wrong and hudge files with strange caracters appears in folders , with anothers sdcard not succeed. So my opinion is to concentrate attention at sdcard, try to use for another productor, i speak from my experience , i hope that you problem is not sdcard! good luck!

It seems you still do not know whether the write to the SD card fails, or the data sampling is not taking place to begin with.

I assume you know that the ESP32 already comes with plenty of flash memory to store months or even years of data. All have at least 4Mb of flash and some have 8mb, 16mb or more, of which only 1mb is needed for the ESP32 program itself.
A good programmer won't need an SD card.

Wawa has a very valid point.

My last project was migrated from a previous one using an Arduino Pro Mini. That has little onboard storage, apart from 1k eeprom. So SDcard was ideal. It worked and I had no problems.

I didn't think to explore what may already be available to use on an ESP32, when migrating to that.
Must check that out. Only lose what might be seen as the convenience of using an SDcard to move the data onto a PC.

There's something called LittleFS for the ESP32 which converts part of the flash memory to a "drive". That might be an option.

But I want to again raise the question of what exactly is happening. We haven't seen your code, but while it could be a problem with the SD card, it could also be a problem with generating the logging entry in the first place.

How can you tell that an entry is missing from the file? Is it the timestamp? Well I'm wondering if you could change you code, just as a test, to have it create all the entries as normal, but only save to the SD card the entries that are too long after the previous entry. Maybe that still doesn't isolate the problem, but maybe you could think about how you would do that in a definitive way.

Also, is there any pattern to the missed entries? By that I mean - does the first missed entry occur at the same elasped time after startup, or could the first one occur at anytime? And thereafter, is there a fixed time between missed entires? Wondering whether the watchdog timer could be involved.

Would you consider posting your code?

So does SPIFFS (and it's not deprecated for an ESP32) and i think there is even another option.

It is not hard to send or receive a file over WIFI using a form from a webserver, either an SD-file or in SPIFFS or LittleFS

[quote="Deva_Rishi, post:16, topic:1448729, full:true"]

I would have to disagree with you here. ('not hard')
I've just looked at a guide for doing that and it seems to be a larger project than the project I'd be incorporating it into.

The Rui Santos two pushbutton example code compiled, uses 55% of program memory of a FireBeetle 2..
I'm totally lost in the http/ html/ java/ stuff/ to get it working. I assume creating a file transfer facility wouldn't be any more simple?

I can't deny, it would be really convenient to run WinSCP or the like, to transfer a file. I use that to get stuff off my android phone, as it's easier than using a USB cable and trying to access it from windows!

That is likely 55% of the allocated partitioned part of the flash.
Default ESP32 partitions are very wasteful (ignoring most of the) flash.
A 4Mb flash could use 1/4 for the program, another 1/4 for OTA updates, 1/4 for storage and the rest ignored.

But I agree. There are not many examples that show you how to partition economically and upload the data OTA (over the air) to a PC. It took me some time/effort to partition an 8MB ESP32-S3 into 1Mb program and 7Mb LittleFS storage. Upload to PC is still above my capability.

That is possible. I managed several years ago and have of course just been incorporating it quite a few projects.

It is just HTML from a webserver, but if you have not created a webserver before at all it is skipping some educational steps to do this.

By now though i wonder why there is a need for an ESP32 at all ? the Wifi capability is usually the feature that is the one that makes one choose an ESP.

A lot of that is comprised of core functions that may not even be used , but adding some functionality to the code will not make the size of the binary grow much at all.

have a look at this thread and
this one which i bookmarked myself.

i have an example for the ESP8266

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266mDNS.h>
#include <ESP8266WebServer.h>
#include <FS.h>   // Include the SPIFFS library

ESP8266WiFiMulti wifiMulti;     // Create an instance of the ESP8266WiFiMulti class, called 'wifiMulti'

ESP8266WebServer server(80);    // Create a webserver object that listens for HTTP request on port 80

File fsUploadFile;              // a File object to temporarily store the received file

String getContentType(String filename); // convert the file extension to the MIME type
bool handleFileRead(String path);       // send the right file to the client (if it exists)
void handleFileUpload();                // upload a new file to the SPIFFS

void setup() {
  Serial.begin(115200);         // Start the Serial communication to send messages to the computer
  delay(10);
  Serial.println('\n');

  wifiMulti.addAP("ssid_from_AP_1", "your_password_for_AP_1");   // add Wi-Fi networks you want to connect to
  wifiMulti.addAP("ssid_from_AP_2", "your_password_for_AP_2");
  wifiMulti.addAP("ssid_from_AP_3", "your_password_for_AP_3");

  Serial.println("Connecting ...");
  int i = 0;
  while (wifiMulti.run() != WL_CONNECTED) { // Wait for the Wi-Fi to connect
    delay(1000);
    Serial.print(++i); Serial.print(' ');
  }
  Serial.println('\n');
  Serial.print("Connected to ");
  Serial.println(WiFi.SSID());              // Tell us what network we're connected to
  Serial.print("IP address:\t");
  Serial.println(WiFi.localIP());           // Send the IP address of the ESP8266 to the computer

  if (!MDNS.begin("esp8266")) {             // Start the mDNS responder for esp8266.local
    Serial.println("Error setting up MDNS responder!");
  }
  Serial.println("mDNS responder started");

  SPIFFS.begin();                           // Start the SPI Flash Files System

  server.on("/upload", HTTP_GET, []() {                 // if the client requests the upload page
    if (!handleFileRead("/upload.html"))                // send it if it exists
      server.send(404, "text/plain", "404: Not Found"); // otherwise, respond with a 404 (Not Found) error
  });

  server.on("/upload", HTTP_POST,                       // if the client posts to the upload page
  []() {
    server.send(200);
  },                          // Send status 200 (OK) to tell the client we are ready to receive
  handleFileUpload                                    // Receive and save the file
           );

  server.onNotFound([]() {                              // If the client requests any URI
    if (!handleFileRead(server.uri()))                  // send it if it exists
      server.send(404, "text/plain", "404: Not Found"); // otherwise, respond with a 404 (Not Found) error
  });

  server.begin();                           // Actually start the server
  Serial.println("HTTP server started");
}

void loop() {
  server.handleClient();
}

String getContentType(String filename) { // convert the file extension to the MIME type
  if (filename.endsWith(".html")) return "text/html";
  else if (filename.endsWith(".css")) return "text/css";
  else if (filename.endsWith(".js")) return "application/javascript";
  else if (filename.endsWith(".ico")) return "image/x-icon";
  else if (filename.endsWith(".gz")) return "application/x-gzip";
  return "text/plain";
}

bool handleFileRead(String path) { // send the right file to the client (if it exists)
  Serial.println("handleFileRead: " + path);
  if (path.endsWith("/")) path += "index.html";          // If a folder is requested, send the index file
  String contentType = getContentType(path);             // Get the MIME type
  String pathWithGz = path + ".gz";
  if (SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)) { // If the file exists, either as a compressed archive, or normal
    if (SPIFFS.exists(pathWithGz))                         // If there's a compressed version available
      path += ".gz";                                         // Use the compressed verion
    File file = SPIFFS.open(path, "r");                    // Open the file
    size_t sent = server.streamFile(file, contentType);    // Send it to the client
    file.close();                                          // Close the file again
    Serial.println(String("\tSent file: ") + path);
    return true;
  }
  Serial.println(String("\tFile Not Found: ") + path);   // If the file doesn't exist, return false
  return false;
}

void handleFileUpload() { // upload a new file to the SPIFFS
  HTTPUpload& upload = server.upload();
  if (upload.status == UPLOAD_FILE_START) {
    String filename = upload.filename;
    if (!filename.startsWith("/")) filename = "/" + filename;
    Serial.print("handleFileUpload Name: "); Serial.println(filename);
    fsUploadFile = SPIFFS.open(filename, "w");            // Open the file for writing in SPIFFS (create if it doesn't exist)
    filename = String();
  } 
  else if (upload.status == UPLOAD_FILE_WRITE) {
    if (fsUploadFile)
      fsUploadFile.write(upload.buf, upload.currentSize); // Write the received bytes to the file
  } 
  else if (upload.status == UPLOAD_FILE_END) {
    if (fsUploadFile) {                                   // If the file was successfully created
      fsUploadFile.close();                               // Close the file again
      Serial.print("handleFileUpload Size: "); Serial.println(upload.totalSize);
      server.sendHeader("Location", "/success.html");     // Redirect the client to the success page
      server.send(303);
    } 
    else {
      server.send(500, "text/plain", "500: couldn't create file");
    }
  }
}

The ESP32 uses slightly different libraries but the general gist and the HTML code are almost the same.

i have my own functions and methods, but they are actually not all that complex if you are used to working with the WebServer.h though they are slightly different for an SD card than a SPIFFS or LittleFS file due to the variation in their implementation.

Regardless, the way to proceed is to first set up a simple webserver and understand how it works.

Transferring a file over HTTP to a web server is not that complicated. You can hard-code the part to "perform HTTP manually", which some examples do; an HTTP/web client library would take a fair amount of space. No need for HTML at all. Probably wouldn't be using Java, and no need for JavaScript -- don't confuse the two: like the difference between potatoes and potato chips. Both tasty in their own way, but with different overlapping applications.

So then there's the matter of running a web server that accepts files for upload. Lots of choices there, maybe too many. Doing a search on "one liner web server that accepts file uploads" returns a few options. If you're on the local LAN, there is less need to screen out random uploads from strangers taking up all the disk space.