Reading from SD card fails

Other Hardware Storage
1

I have a SD card connected to an Arduino Nano ESP32. I am able to write to it but not read from it. Here is a sketch that reproduces the issue:

#include <SD.h>
#include <SPI.h>

File myFile;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  delay(2000);
  if (!SD.begin(10)) {
    Serial.print("fail");
    while (!SD.begin(10)) {delay(100);}
  }
  myFile = SD.open("/sys/ram.txt", FILE_WRITE);
  myFile.seek(0);
  Serial.print(myFile.read());
}

void loop() {
  // put your main code here, to run repeatedly:

}
2

I have a vague memory that opening a file for writing erases the existing contents. If that's the case, the there will not be any data to read.

I think you need to open the file in append mode to be able to read existing contents.

1 Like
3

This seems to disagree

Screenshot 2025-02-02 at 12.55.20
Screenshot 2025-02-02 at 12.55.201048×362 48.7 KB

4

I'm on a small screen at the moment so looking at code is a PITA!

If the FILE_WRITE is the same as the write mode in the c fopen function, then I think it creates a new empty file or truncates an existing file.

5

I added a line to write before reading then I closed the file. It says it wrote 6 bytes and failed to read (returns -1) but opening it on my computer shows it being full. I tried to close the file then reopen the file in read mode after writing but it returns 98 instead of b. It should be able to read in write mode and I don't understand why it returns 98. 98 might be something encoded but it shouldn't print that.

6

That's good news: consult any ASCII chart.

read returns an int because it has to be able to return 257 different possible values

  • 0 to 255 for a valid byte
  • -1 if there was nothing to read
7

To create a file inside a directory on the SD card, you first need to create the directory.

SD.mkdir("/sys");

Then you need to write something inside it and then close the file.

To read the content, you open the file, read it and print (or save) the content.

See if the example below meets your needs.

#include <SD.h>
#include <SPI.h>

File myFile;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  delay(200);
  if (!SD.begin(10)) {
    Serial.print("fail");
    while (!SD.begin(10)) {
      delay(100);
    }
  }
  SD.mkdir("/sys");
  myFile = SD.open("/sys/ram.txt", FILE_WRITE);
  myFile.println("1.2.3");
  myFile.close(); // close the file:

  myFile = SD.open("/sys/ram.txt");
  while (myFile.available()) {
    Serial.write(myFile.read());
  }
  myFile.close(); // close the file:
}

void loop() {
  // put your main code here, to run repeatedly:

}
8

The directory has already been created inside the computer.

9

ANy file processing has one pointer to where you are in that file. Writing updates that pointer to the next byte. Then reading will read from that new pointed to byte, which is now beyond the byte written.

10

Why not? You told it to.

In the IDE, if you type (in a function, where it would work)

Serial.println(

the close parenthesis should auto-complete (if not, press Backspace to delete the open parenthesis and re-type it) and you'll get a little popup

01/15 println() -> size_t

There are 15 overloaded variations of the println function. This first one takes no arguments. If you press the Down arrow, you can scroll through the rest. They include

02/15 println(char) -> size_t
03/15 println(const char *) -> size_t
07/15 println(int, int = DEC) -> size_t
08/15 println(double, int = 2) -> size_t
11/15 println(unsigned int, int = DEC) -> size_t
13/15 println(unsigned long, int = DEC) -> size_t

They all return size_t: how many "characters" were printed, including the two for the CR+LF when println is used instead of print. Try it

  auto x = 456.3;
  Serial.println(Serial.println(x, 5));

Which overloads got invoked?

  1. What type is x? 456.3 is a floating point literal without the f suffix, so that's a double. For overload #8, the second argument defaults to 2 decimal places, but 5 are specified. So
    • 456: 3 characters
    • .: 1
    • 5 decimal places
    • 2 for CR+LF
  2. size_t is unsigned (you can't have a negative size) and on Nano ESP32, it's an alias for unsigned long, which is overload #13.
    • When passing an integer type, unsigned or not is an important distinction. The same bits for a signed -1 are also some giant number when unsigned.
    • The second argument for integer types is the radix: DEC is a macro for 10, HEX is a macro for 16 to print in hexadecimal

As I said earlier, read() with no arguments returns an int, either a code point or -1. If you do

that invokes overload #7: it prints that number, not what you were apparently expecting, the corresponding char. To see that, you can assign to a char variable and print that, or do a cast -- a static_cast in this case -- so that the appropriate overload is invoked.

11

I changed the read command to

char c = myFile.read();
Serial.print(c);

and it prints b instead of 98 (which is what I want). I tested with reading in FILE_WRITE mode but that returns �. Why is this?

12

What is "that", the Serial Monitor?

Short answer: you're probably reading at EOF (end-of-file)

Long answer: the Serial Monitor supports Unicode, via UTF-8. Any byte that's not part of a valid UTF-8 sequence for a given code point U+uvwxyz prints a �, which is the replacement character U+FFFD

First code point Last code point Byte 1 Byte 2 Byte 3 Byte 4
U+0000 U+007F 0yyyzzzz
U+0080 U+07FF 110xxxyy 10yyzzzz
U+0800 U+FFFF 1110wwww 10xxxxyy 10yyzzzz
U+010000 U+10FFFF 11110uvv 10vvwwww 10xxxxyy 10yyzzzz

Each byte must have a fixed zero bit

  • If the high (first) bit is zero, then the remaining bits encode ASCII, which is a seven-bit character set
    • The simplest case is when the text is all-ASCII: one byte, one character
  • For a two-byte sequence, the first byte is two 1-bits, followed by a 0-bit
    • This makes UTF-8 encoding self-synchronizing, since you can quickly find the start of the next or previous multi-byte sequence, or if you're not in one
    • The same rule applies for three- and four-byte sequences: three or four 1-bits, followed by a 0-bit
  • The remaining bytes in the sequence have a single 1-bit and 0-bit to start

In addition to invalid multi-byte sequences, these rules mean that several bit patterns are never valid for a byte in UTF-8, including

  • five or more 1-bits to start
  • for the start of a two-byte sequence, 110xxxyy, the lowest code point is U+0080. Therefore, if xxx is all-zero, then yyyy must be at least 8. That means of the two high bits in the first byte, the first y cannot be zero.
    • 110'000'00 is valid only with Modified UTF-8 when followed by 10'00'0000 to represent NUL U+0000
    • 110'000'01 is always invalid as overlong

So then the question is whether your file is UTF-8 text (and ASCII is a strict subset). If not -- binary data, text in another encoding -- then you cannot simply (and repeatedly) read a byte and print it, and expect to see the text as intended in the Serial Monitor.

More likely though, you read() at EOF and got -1, which as char is all-1-bits and invalid UTF-8.

Some UTF-8 demo code, including the top half of Latin-1:

void setup() {
  Serial.begin(115200);
  char c = -1;
  Serial.print("-1: ");
  Serial.println(c);
  Serial.print("section: ");
  char section[] = "§";  // U+00A7
  Serial.println(section);
  Serial.print("alarm clock: ");
  char alarmClock[] = "⏰";  // U+23F0
  Serial.println(alarmClock);
}

void loop() {
  // https://en.wikipedia.org/wiki/UTF-8#Description
  // U+0080-U+07FF	110xxxyy	10yyzzzz
  static uint8_t u1 = 0b110'000'11;  // U+00C? -> yyyy: 1100
  static uint8_t u2 = 0b10'00'0000;  // U+00F? -> yyyy: 1111
  if (u2 < 0b11'00'0000) {
    Serial.print("U+00");
    Serial.print(((u1 & 0x3) << 2) + ((u2 & 0x30) >> 4), HEX);
    Serial.print(u2 & 0xF, HEX);
    Serial.print(" ");
    Serial.print(u1, HEX);
    Serial.print(" ");
    Serial.print(u2, HEX);
    Serial.print(": ");
    char buf[] = {u1, u2++, 0};
    Serial.println(buf);
  }
}
13

Does that mean that I am reading the end of the file even though I have

myFile.seek(0);

before reading? Does this mean seek() doesn't work?

14

I checked the position after seeking and it's zero (meaning it's at the start of the file). Does this mean I have to close and open the file every time I need to read?

15

Changing between modes is not efficient and switching to write mode erases all data. If I were to use this method, I need to find out how to preserve the file's data.

16

Try this

  myFile = SD.open("/sys/ram.txt", "a+");

The critical factor is you're using ESP32. The file mode macros in the platform-specific lower-level FS library are

#define FILE_READ   "r"
#define FILE_WRITE  "w"
#define FILE_APPEND "a"

In contrast, Uno (for example) uses the "root level" generic SD library, which has only two modes

#define FILE_READ O_READ
#define FILE_WRITE (O_READ | O_WRITE | O_CREAT | O_APPEND)

where WRITE includes READ (and APPEND). But on ESP32, they're entirely separate. With "w" alone, read() always returns -1. Drill in far enough, and the implementation uses good old fopen

  if (!stat(temp, &_stat)) {
    //file found
    if (S_ISREG(_stat.st_mode)) {
      _isDirectory = false;
      _f = fopen(temp, mode);

There's a nice table at std::fopen - cppreference.com that tells you what you need; an excerpt

File access
mode string		 Meaning Explanation Action if file
already exists
w write Create a file for writing destroy contents
r+ read extended Open a file for read/write read from start
w+ write extended Create a file for read/write destroy contents
a+ append extended Open a file for read/write write to end

"r+" fails if the file does not exist, so you probably want "a+". As mentioned by @markd833 a week ago, either "w" or "w+" erases an existing file. Sadly the ESP32 FS library makes no mention of these extended modes, nor provides any support for them. So you need to manually spell the mode string.

Apparently on ESP32, there's a poorly named optional third argument: create

File open(const char* path, const char* mode = FILE_READ, const bool create = false);

which when enabled

  ////file not found but mode permits file creation and folder creation
  if ((mode && mode[0] != 'r') && create) {

will loop through and create each path segment. Didn't try it.

17

It works! I am able to read and write. I've been using w+ at the start to clear it and it fixes another problem where it creates a big file. A new problem has arisen where w+ clears the file but when something gets printed, the old text comes back.

18

I'd hazard a guess that you are reading past the end of the file and seeing old data. Off the top of my head - because it's been a while now - when you "erase" the file contents, all that is happening is a file size in the depths of the SD card library gets set back to 0. The old data still exists on the SD card until it's over-written. I would have thought that your read function call would return -1 or something similar to indicate an error.

19

I did tests and found out it's writing to the end of the file when "erasing".

20

I fixed it by adding changing it to:

myFile = SD.open("/sys/ram.txt", "w");
myFile.close();
myFile = SD.open("/sys/ram.txt", "a+");

The size command doesn't return a big number now.