Serial data issues when writing to SD - ESP32

Projects General Guidance
1

Hello! This is my first post, so... sorry in advance if everything is all over the place. :grinning:

For the past few weeks I have been developing on the ESP32-S3-USB-OTG board using the ESP-IDF implementation for Arduino IDE . My project involves reading incoming serial data from a sensor at 30k CPS and storing it to and SD card (essentially a pretty fast datalogger). In order to simulate the data coming into the board, I have been using Realterm to send a .txt file with sample data at the same speed and same size of 252 bytes. However, I noticed that most of the data returns with significant errors, and so I wrote some debug code to figure out if the UART was causing issues or if the SD card was causing issues.

Through my experimentation I found that, by itself, the UART can keep up with the high baudrate and return data correctly without errors (tested by the error counter implemented in the code as well as by sending the data over WiFi). However, issues with reading the data occurs when adding the code to append to the SD card (which is an UHS class 1 card).

I have tried different implementations of reading data over the Serial port (I'm aware that there are some issues involving Strings, sadly different implementations without the usage of Strings still caused issues). It seems that no matter the implementation, the errors still occur as long as the SD_MMC append function is called. I have also tried the SdFat and SD libraries with similar results.

I was wondering if someone has encountered something similar, and if someone can offer some guidance as to why this error might be occurring and what I can do to improve my project's functionality.

Thanks! (below is my code and the data received from the SD card - which received about 289 errors)

CODE:

#include <FS.h>
#include <SD_MMC.h>

const char* filename = "/gd.txt";                            // File name prefix for SD storage
uint16_t data_caught_num = 0;                                // amount of errors received via serial

void appendFile(fs::FS &fs, const char * path, const char * message){   
    //code grabbed from the examples in the ESP32 SD_MMC library
    File file = fs.open(path, FILE_APPEND);
    if(!file){
        Serial.println("Failed to open file for appending");
        return;
    }
    if(!file.print(message)){
        Serial.println("Append failed");
    }
    file.close();
}

void setup() {                                                // begins Serial and SD_MMC 
  Serial.begin(460800); 
  SD_MMC.begin(); 
}

void loop() {
  if(Serial.available() > 0) {                                // if receiving serial data from UART
    String raw_data = Serial.readStringUntil('\x0D');         // read incoming serial data as a string until terminator

    if(raw_data.length() < 252) {                             // debug code to catch errors where the data is less than the expected 252 characters
      data_caught_num += 1;
    }
    
    const char* raw_dat = raw_data.c_str();                   // converts string and appends to SD card
    appendFile(SD_MMC, filename, raw_dat);                   

  } else {
    Serial.println("ERROR DATA COUNT:");
    Serial.println(data_caught_num - 1);                     // minus 1 because the last line in the sent data is half of a line for testing purposes, so this line is just omitted in the error tally.
  }
}

When writing the mock data to the SD card, some of the data comes back bad (lines 3, 4, 6) while others come back good (1, 2, 5, 7).

SENSOR DATA WRITTEN TO SD:

00185.54,00184.72,00188.30,00216.77,00173.88,00176.04,00169.01,00170.61,00172.19,00173.55,00180.39,00196.65,00180.23,00188.19,00194.35,00193.42,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000000,00000200
00185.75,00184.98,00188.47,00216.45,00172.94,00174.74,00168.45,00170.24,00172.55,00174.86,00180.87,00196.67,00179.74,00186.53,00192.71,00193.28,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000000,00000208
00185.80,00184.81,00188.33,00216.62,00173.93,00176.23,00170.00,00171.28,00172.97,00173.51,00
00186.72,00185.48,00188.34,00215.47,00172.20,00174.87,00170.63,00172.96,00174.47,00175.41,00179.91,00195.69,00179.44,00187.63,00194.25,00195.89,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000000300
00186.52,00185.42,00188.40,00215.93,00171.98,00174.50,00169.64,00171.58,00173.97,00175.65,00180.38,00195.97,00179.45,00186.91,00193.09,00194.88,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000000,00000308
00186.07,00184.82,00188.46,00216.46,00173.17,00175.50,00169.34,00170.82,00173.02,00174.04,00180.36,00196.30,00180.00,00188.04,00194.09,00193.96,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0009.82,00195.80,00180.27,00189.08,00195.13,00194.64,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000000,00000325
00186.27,00185.00,00187.90,00215.51,00172.69,00175.66,00171.00,00173.23,00174.40,00173.84,00179.55,00195.14,00179.73,00188.38,00195.19,00195.94,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000000,00000333
Writing to a R/W Register
2

Welcome to the forum.

When speed and timing is a issue, then don't use the Serial.readStringUntil() and String there.
Can you tell more about the data ? I suppose it is 460800 baud. And the packet is 252 bytes ? How much pause is there between the packages ? Enough time to write the 252 bytes ?

Here is a good guide for a ESP with a SD card: https://randomnerdtutorials.com/esp32-microsd-card-arduino/
I had SD card trouble in the past, so now I have a bunch of older 8GB µSD cards that I'm sure that work, and I format them with the tool that they mention.

Their datalogger uses the "SD" and not the "SDD_MMC": https://randomnerdtutorials.com/esp32-data-logging-temperature-to-microsd-card/.
Can you try to make a sketch with the "SD" library ?

Even though FAT32 is old, it is not easy. It needs to update the begin of the SD card when something is written, that takes time.

3

Thanks for the welcome.

The data is coming in a constant stream of 252 bytes/line with 120 lines/second (about 30k bytes/second), so I just covered my bases and went with a standard 460800 baudrate (which I tested by itself and it correctly read and gave back the data). I previously ran a benchtest on the board for the SD card and it was supposedly writing at a speed of ~3 MB/sec, which while on the low side, should still be enough to keep up with the incoming serial data.

I should also mention that I am using the built in SD slot on my dev board instead of an external one. I have also used both the windows formatter and the proprietary(?) one at sdcard.org, with no results.

I tried using the SD library instead of SD_MMC shown in the first link, however, the results were the same.

I'm not sure what you meant by "pause" between packets, but I recorded the time in ms for the main part of the loop to execute and I returned the average (seen at the end of this reply). The returned value was 13.03 ms, which may be the root of my issue since the Rx buffer might be filling up quicker than I can read from it. If that's the case, is it possible to have the SD card write at these speeds?

#include <FS.h>
#include <SD_MMC.h>

const char* filename = "/gd.txt";                            // File name prefix for SD storage
uint16_t data_caught_num = 0;                                // amount of errors received via serial
float t1 = 0;
float t2 = 0;
uint32_t count = 0;
float t_total = 0;

void appendFile(fs::FS &fs, const char * path, const char * message){   
    //code grabbed from the examples in the ESP32 SD_MMC library
    File file = fs.open(path, FILE_APPEND);
    if(!file){
        Serial.println("Failed to open file for appending");
        return;
    }
    if(!file.print(message)){
        Serial.println("Append failed");
    }
    file.close();
}

void setup() {                                                // begins Serial and SD_MMC 
  Serial.begin(460800); 
  SD_MMC.begin(); 
}

void loop() {
  if(Serial.available() > 0) {                                // if receiving serial data from UART
    t1 = millis();
    String raw_data = Serial.readStringUntil('\x0D');         // read incoming serial data as a string until terminator

    if(raw_data.length() < 252) {                             // debug code to catch errors where the data is less than the expected 252 characters
      data_caught_num += 1;
    }
    
    const char* raw_dat = raw_data.c_str();                   // converts string and appends to SD card
    appendFile(SD_MMC, filename, raw_dat);                   
    t2 = millis();
    t_total += t2 - t1;
    count += 1;
  } else {
    Serial.println("ERROR DATA COUNT:");
    Serial.println(data_caught_num - 1);                     // minus 1 because the last line in the sent data is half of a line for testing purposes, so this line is just omitted in the error tally.
    Serial.println("Average Time For One Loop");
    Serial.println(t_total / count);
  }
}
4

Performing an open and close for each line in the file is sub-optimal when it comes to SD sector size.

May be you could study some of the data logger examples from the SdFat library

5

The data comes in continuously ? not in bursts ?

It feels indeed as if the data can not be written fast enough.

How do you start and how do you stop the file ?
Opening a file for append for only a lousy 252 bytes is too much FAT32 administration trouble. Can you open a file, store all the Mega/Giga bytes that you need and then close the file ?
(while I was writing this, J-M-L wrote the same)

The data is read as bytes and the data is stored as bytes. Using the 'String' object in between is not optimal. It uses the heap a lot. You have to optimize that as well.

6

I've took a look at the SdFat library examples a few days ago (however I stopped since most of the relevant ones seemed daunting ~1000 lines), but I will check again to see if there is any relevant datalogging practices that could help.

Yes, the data comes in continuously.

In a previous iteration of the code I tried opening the file once and then closing it after there was no more serial data in the buffer, but sadly the issue still occurred.

I have also previously tried storing the read data into an array of chars through the Serial.read() call, and similar issues occurred.

EDIT: Just to try it again, I changed the code to only open the file once(ish) and instead of using Serial.readStringUntil(), I tried using Serial.read() and Serial.readBytesUntil(). With the readBytesUntil(), it still produces a significant amount of error (more so than the readStringUntil, however that might be due to my implementation) and the read() produces more errors than both (once again, my implementation is probably poor). The way the code is structured now is reportedly running at an average of 8.39 ms, down from the 13 I was recording before.

This is the current code:

#include <FS.h>
#include <SD_MMC.h>

const char* filename = "/gd.txt";                            // File name prefix for SD storage
uint16_t data_caught_num = 0;                                // amount of errors received via serial
float t1 = 0;
float t2 = 0;
uint32_t count = 0;
float t_total = 0;
char buff[252];

File file;

void setup() {                                                // begins Serial and SD_MMC 
  Serial.begin(460800); 
  SD_MMC.begin();
}

void loop() {
  if(Serial.available() > 0) {                                // if receiving serial data from UART
    t1 = millis();

    uint16_t data_length = Serial.readBytesUntil('\x0D', buff, 252);
    
    if(data_length < 252) {                             // debug code to catch errors where the data is less than the expected 252 characters
      data_caught_num += 1;
    }
    
    file = SD_MMC.open(filename, FILE_APPEND);
    file.print(buff);
                       
    t2 = millis();
    t_total += t2 - t1;
    count += 1;
  } else {
    Serial.println("ERROR DATA COUNT:");
    Serial.println(data_caught_num - 1);                     // minus 1 because the last line in the sent data is half of a line for testing purposes, so this line is just omitted in the error tally.
    Serial.println("Average Time For One Loop");
    Serial.println(t_total / count);
    file.close();
  }
}
7

What makes that you start the recording of the data ?
What makes that you stop the recording of the data ?

If you start when the parrot of the neighbor says: "start" and you stop when the 16GB SD card is filled with the same percentage as the humidity, then at least I have an idea.

8

Ideally, it should start when the UART starts receiving data, and it should stop when the SD card is full (or if there is no more serial data to be read).

9 
  // discard any random input
  while (Serial.read() >= 0) {}
} // setup

void loop() {
  if (Serial.available()) {
      c = Serial.read();
      buffer[bufferidx++] = c;  // Store character in array and increment index
      if (c == '\r' || (bufferidx >= BUFFSIZE-1)) {
        buffer[bufferidx] = 0; // terminate it with null at current index
        logfile.write((uint8_t *) buffer, (bufferidx + 1));
        logfile.flush();
        bufferidx = 0;     // reset buffer pointer
      }
  }
}

MY preference from the old days is to come out of setup() knowing that any partial serial has gone into the bit-bucket; that is, internal serial buffer is empty.

Then it is a matter to read character-by-character into the buffer until the CR/LF is caught, replace that character with null, and go about writing to the SD, flushing the SD buffer before dealing with serial (buffered by interrupts into the serial buffer ... which can be extended.)

In the snippet, nothing is checking for the close-file state, such logic needs to be added. I do not know if the ".flush" is even in the current SD library, but a similar method surely is available.

Yep: SD - flush() - Arduino Reference

10

Thanks for the tips!

I tried something similar in a previous attempt to get Serial.read() working (I didn't know you could increment inside the buffer index!). I tried implementing your code just to see what would happen and pretty much the same thing happened as in my previous attempts (I'm not sure how if the .flush() actually works, since no data would be written to the SD card when this was called, so I ended up removing it and just leaving in a file.close() at the end - this happened with both SD.h and SD_MMC.h).

Here is some of the data written to the sd card:

00185.75,00184.98,00188.47,00216.45,00172.94,00174.74,00168.45,00170.24,00172.55,00174.86,00180.87,00196.6000000,00000000,00000000,00000000,00000000,00000000,000000000,00000633
 
00188.44,00185.14,00189.89,00214.09,00174.03,00176.49,00172.07,00173.31,00173.05,00172.97,00178.44,00193.38,00179.93,00188.03,00195.21,00196.27,00000000,00000000,0000002.05,00197.38,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000 00,00000000,00000000,000000000,00001091

and here is the code with pretty much the same setup as the one you gave.

#include <FS.h>
#include <SD_MMC.h>

const char* filename = "/gd.txt";                            // File name prefix for SD storage
float t1 = 0;
float t2 = 0;
uint32_t count = 0;
float t_total = 0;
uint8_t buff_i = 0;
char buff[252];
char c;

File file;

void setup() {                                                // begins Serial and SD_MMC 
  Serial.begin(460800); 
  SD_MMC.begin();
  while(Serial.read() >= 0) {}
}

void loop() {
  if(Serial.available()) {                                // if receiving serial data from UART
    t1 = millis();
    
    file = SD_MMC.open(filename, FILE_APPEND);
    
    c = Serial.read();
    buff[buff_i++] = c;
    if(c == '\r' || (buff_i >= 251)) {
      buff[buff_i] = 0;
      file.write((uint8_t *) buff, (buff_i + 1));
      buff_i = 0;
    }

    t2 = millis();
    t_total += t2 - t1;
    count += 1;
  } else {
    Serial.println("Average Time For One Loop");
    Serial.println(t_total / count);
    file.close();
  }
}

On the bright side, the loop only took on average 1.33 ms with is significantly faster than any of my previous attempts, but only 7 kB out of the ~277 kB was stored, but this could just be the way I am closing the file or due to the size of my buffer (which I am 99% sure it is 252 characters unless there is some hidden characters my text reader isn't seeing in a line).

11

Arthritis prevents me from patting myself on the back, but I'll smile on the speed increase.

Now, to your issue(s.) The serial input has an interrupt driven buffer, default is in the core files and generally is not recommended to modify. Writing a file record copies to a buffer inside the SD card. The file.flush() method is two-fold, it forces all of the SD class to write and informed the SD hardware controller to prepare to close the file, but stops short of actually closing it. File.close() calls file.flush() first and then closes the file.

My guess is that you need to enlarge your input buffer to 255 (or larger.) Write a sketch that can run a 512 buffer and with iteration, print the size if it is larger than the previous size. Run the program for a few hours looking for abnormalities.

The 2nd potential is addressed in this article. Remember, the ESP32 core is different and addressed here.

Note: changing the core code is a super-sensitive issue in the forum. I have not looked recently to determine if the buffer_size can be easily modified. Discussed here.

Ladyada (Adafruit) in her GPS library actually creates a double-buffer for serial input and most GPS use 9600 BAUD but recent units support up to 57600.

Afterthought:
You are running on a dual cure chip. I have done some freeRTOS work on ESP32 years back; Arduino is on core_1 and RF and Espressif stuff on core_0.
I attached my old dual-core play code.

DualCore.zip (2.2 KB)

12

Cross post here: Writing to a R/W Register

13

I'm happy about the speed increase too!

What's bizarre about file.flush() is that when I called it in the code, I noticed that the time it took to run was 0.02 ms. I popped the sd card into my computer to look at the .txt file it wrote, and there was no .txt file to be found. So I'm not entirely sure if file.flush() is a valid call anymore (maybe it doesn't actually work anymore, or my board just doesn't register it?).

I also thought about enlarging my Rx buffer (since I believe it's at 128 bytes by default). I looked at this awhile back and I tried to edit the board files and set the Serial.setRxBufferSize(), but neither seemed to work (I probably followed the instructions wrong horribly or something, so that might be an avenue to pursue again). In my readings of the ESP32-S3 technical datasheet, I came across a r/w register that extends the Rx0 buffer to take up the space in memory of the Rx1 and Rx2 buffers in the board which remains unused. I couldn't find a way to actually write to this register (which I express a bit further in my cross-post referenced above - sorry about that). Since writing to the register seems impossible without going to ESP-IDF (which I would rather never have to do), I will consider attempting to change the core code, and do a clean install if it doesn't work.

I haven't heard of this GPS library, so I will also look into this double-buffer to see if it can be applied to my code.

It's also funny that you mention using two cores, since that is actually what I attempted to do most recently. In the implementation shown below, I use my own awfully coded circular buffer "library" (just to give myself some more control over the circular buffer library I was previously using - GitHub - rlogiacco/CircularBuffer: Arduino circular buffer library). This attempt at using two cores and my own circular buffer probably ended up causing more issues than it solved, but it's good to see that my thought process of running on two cores might have been a good first step at solving this problem (if I end up not being able to change my Rx buffer size). It is also possible I misinterpreted some of the 2nd core startup parameters, so thank you so much for sending your code.

my pitiful attempt at 2 cores and the most grotesque code you have ever seen:

#include <FS.h>
#include <SD_MMC.h>

TaskHandle_t Core2Task;

String FileName = "/gd.txt";                                   // File name prefix for SD storage
bool writing = false;
uint16_t read_i = 0;
uint16_t write_i = 0;
const uint16_t ring_buffer_len = 60;
String ring_buffer[ring_buffer_len];

String circular_buffer(bool reading, String data_to_write) {
  String data_read = "0";
  if(reading == true && (read_i != write_i || writing == false)) {
    if(read_i > (ring_buffer_len - 1)) {
      read_i = 0;
    }
    if(ring_buffer[read_i] != "0") {
      data_read = ring_buffer[read_i];
      ring_buffer[read_i] = "0";
      read_i += 1;
    }
  } else if(reading == false) {
    if(write_i > (ring_buffer_len - 1)) {
      write_i = 0;
    }
    writing = true;
    ring_buffer[write_i] = data_to_write;
    write_i += 1;
    writing = false;
  }
  return data_read;
}

void appendFile(fs::FS &fs, const char * path, const char * message){
    File file = fs.open(path, FILE_APPEND);
    if(!file){
        Serial.println("Failed to open file for appending");
        return;
    }
    if(!file.print(message)){
        Serial.println("Append failed");
    }
    file.close();
}

void CoreTwoCode(void*) {
  Serial.println(xPortGetCoreID());
  while(1) {
    if(Serial.available() > 0) {
      String incoming_data = Serial.readStringUntil('\x0D');
      circular_buffer(false, incoming_data);
    }
  }
}

void setup() {
  Serial.begin(460800);                                       // set serial baud rate
  xTaskCreatePinnedToCore(CoreTwoCode, "Core2Task", 10000, NULL, 0, NULL, 0);
  Serial.println(xPortGetCoreID());
  SD_MMC.begin();
}

void loop() {
  if(read_i != write_i && ring_buffer[read_i] != "0") {                                // if receiving serial data from UART
    String raw_data = circular_buffer(true, "0");
    if(raw_data != "0") {
      raw_data += '\x0D';                                       // adds back terminator character to the data
      const char* raw_dat = raw_data.c_str();
      appendFile(SD_MMC, filename, raw_dat);                    // converts the data to a char and appends cus im bad at coding
    }
  }
}
14

The 2-cores are started separately. Core_0 starts, freeRTOS is initialized at some point, and then the initialization of Core_1 and Arduino code given a task controlled by freeRTOS.

Review the answer given here:

c++ - How To Increase RX Serial Buffer Size for ESP32 library Hardwareserial (Platform IO) - Stack Overflow


....

 #define BAUD_RATE  115200
 #define SERIAL_SIZE_RX  1024    // used in Serial.setRxBufferSize()

setup(){
  Serial.begin(BAUD_RATE);
  Serial.setRxBufferSize(SERIAL_SIZE_RX);
....
}
15

That model is RISC-V cpu architecture... totally different animal from Tensilica ESP-32.

Windows needs a file to be properly closed.

Consider an interrupt driven push-button to unmount the SD... the interrupt needs to set a (volatile) bool flag that is inspected in loop which would call file.close(). Then the SD can be removed.

I'm unsure if the latest Linux build respects the file closed status of a file on SD.

In the old DOS days, I think Peter Norton's disk utilities would allow one to inspect a floppy file that was still open.

16

Nah. It's the -Cn chips that are RISC-V. The -S3 is a dual core version of the chip with USBOTG and BT5. ESP32 - Wikipedia

Yeah, I could wish for better naming :frowning:

17

What memory size are your SD cards ?

18

Thanks! Too many models to juggle in my old head without
validating each one.
ESP32-S3 Wi-Fi & Bluetooth 5 (LE) MCU | Espressif Systems

From the Espressif page:

ESP32-S3 is a dual-core XTensa LX7 MCU, capable of running at 240 MHz. Apart from its 512 KB of internal SRAM, it also comes with integrated 2.4 GHz, 802.11 b/g/n Wi-Fi and Bluetooth 5 (LE) connectivity that provides long-range support. It has 45 programmable GPIOs and supports a rich set of peripherals. ESP32-S3 supports larger, high-speed octal SPI flash, and PSRAM with configurable data and instruction cache.

The Cx that @westfw corrected me is detailed here:
Introducing ESP32-C3 | Espressif Systems

ESP32-C3 is a single-core, 32-bit, RISC-V-based MCU with 400KB of SRAM, which is capable of running at 160MHz. It has integrated 2.4 GHz Wi-Fi and Bluetooth 5 (LE) with a long-range support. It has 22 programmable GPIOs with support for ADC, SPI, UART, I2C, I2S, RMT, TWAI, and PWM.

19

Sorry, I think I meant task startup parameters (not core startup).

I previously tried using .setRxBufferSize(256), 512, 1024, etc. with no results. (I'm not entirely sure if this is true, but I believe the .setRxBufferSize only works for the USB host CDC and not UART). I have tried to get the USB host working since I believe it would solve a lot of my problems but I could never seem to get it to detect a peripheral I was plugging in when using the ESP32 core USB host example with all the pins configured.

I'll try binding the file.close() functionality to a pushbutton to see if that works with file.flush().
Edit: This seemed to solve the issue with file.flush() and seems to be slightly faster than closing the file, however, the file still remains with a fraction of the data (7 kB) and messed up line formatting/missing data as before.

20

Here is an Amazon link to the exact SD Card I am using: