Seeking to end of file creates large file

Other Hardware Storage
1

Hi. I am working on this piece of code:

void mkVar(String varn, String val, String t) {
  varFile.seek(varFile.size());
  varFile.println(varn + "," + val + "#" + t);
}

It is supposed to go to the end of a file and print some stuff but I noticed that the Arduino was taking a long time to go past this piece of code and when I checked the file via my computer, it was 0,99 GB. Before this code, it deletes the file then reopens it to clear the file. I was doing testing to find out the issue and I printed the size of the file in the Serial then stopped the code to check the file on my computer. The number it printed in Serial was 1070523488 but on my computer it was empty. If I deleted the file before running the sketch, the Serial printed 0. I do not understand why the size was 1070523488 because the file was cleared and I think it created a big file because it tried to seek to an number outside of the file.

Is there another way to seek to the end of the file or is there a way to fix this?

2

When you open the file after having deleted it and before writing anything to it, which is what you describe you are doing, what do you expect the size of the file to be ?

1 Like
3

0 as there shouldn't be anything inside.

4

If you open the file in APPEND mode, then it should always just write to the end of the file. seek would be unnecessary, and might not work at all.

What file system is this? SD? LittleFS? SPIFFS?

Where is the file, such that you can get its size from both the MCU and the computer at the same time?

1 Like
5

If you open the file in APPEND mode, then it should always just write to the end of the file. seek would be unnecessary, and might not work at all.

There are things that can be between the open command and this function that can change where the "cursor" is.

What file system is this? SD? LittleFS? SPIFFS?

I am using SD.

Where is the file, such that you can get its size from both the MCU and the computer at the same time?

It is in a micro SD card. So the Arduino and computer can't read it at the same time.

6

Which SD library? If you compile with Verbose output (and everyone should), at the end you should see something like

Using library SD at version 1.3.0 in folder: /Users/kenb4/Library/Arduino15/libraries/SD

On this Mac, it's the generic SD library baked in at the IDE/CLI level, and neither board- nor library-specific. Anyway, if you Go to Definition on SD.open, it takes you to libraries/SD/src/SD.h

File open(const char *filename, uint8_t mode = FILE_READ);

The default mode is FILE_READ. Drilling in further

#define FILE_READ O_READ
#define FILE_WRITE (O_READ | O_WRITE | O_CREAT | O_APPEND)

FILE_WRITE includes O_APPEND, which is listed among the others in libraries/SD/src/utility/SdFat.h

// use the gnu style oflag in open()
/** open() oflag for reading */
uint8_t const O_READ = 0X01;
/** open() oflag - same as O_READ */
uint8_t const O_RDONLY = O_READ;
/** open() oflag for write */
uint8_t const O_WRITE = 0X02;
/** open() oflag - same as O_WRITE */
uint8_t const O_WRONLY = O_WRITE;
/** open() oflag for reading and writing */
uint8_t const O_RDWR = (O_READ | O_WRITE);
/** open() oflag mask for access modes */
uint8_t const O_ACCMODE = (O_READ | O_WRITE);
/** The file offset shall be set to the end of the file prior to each write. */
uint8_t const O_APPEND = 0X04;
/** synchronous writes - call sync() after each write */
uint8_t const O_SYNC = 0X08;
/** create the file if nonexistent */
uint8_t const O_CREAT = 0X10;
/** If O_CREAT and O_EXCL are set, open() shall fail if the file exists */
uint8_t const O_EXCL = 0X20;
/** truncate the file to zero length */
uint8_t const O_TRUNC = 0X40;

The file offset shall be set to the end of the file prior to each write. I don't have an SD handy, so I tried this in the simulator

  File textFile = SD.open("wokwi.txt", FILE_WRITE);
  Serial.println(textFile.position());
  char buf[20];
  textFile.seek(0);
  buf[textFile.read(buf, sizeof(buf) - 1)] = 0;
  Serial.println(buf);
  textFile.write("first line\n");
  Serial.println(textFile.position());
  buf[textFile.read(buf, sizeof(buf) - 1)] = 0;
  Serial.println(buf);
  textFile.seek(0);
  buf[textFile.read(buf, sizeof(buf) - 1)] = 0;
  Serial.println(buf);
  textFile.write("second line\n");
  textFile.close();

  // Example of reading file from the card:
  textFile = SD.open("wokwi.txt");

The file starts with a line of text. With or without the seek, the new lines were added to the end of the file.

  • when opened to O_APPEND, the initial position is at the end of the file
  • therefore to read anything, first seek the beginning
  • writing appends automatically, updating the position to the new end of file
  • the second read gets nothing
  • the third read gets the first line again, plus some of the newly added line

So if you are only appending when you are writing, O_APPEND should work. If you are writing elsewhere, then you have to come up with your own combination of modes, and you'll need to keep an eye of where you seek, and make sure it is not some crazy location like 1070523488. If that happens, then the problem is where that number came from.

7

I have tested multiple things and only when the file is deleted on the computer, is the file size 0. Even though the code deletes the file to clear it, when you don't delete it off the computer, it returns 1070523488. When it returned 1070523488, I stopped the program to find the size is 0.

8

This is File::size returning 1070523488?

You didn't say which SD library you're using. In particular, ESP32 has its own. Each library has its own way to seek relative to the end of the file

  • f.seekEnd()
  • f.seek(0, SeekEnd)

Can you run CHKDSK or the equivalent on the SD (on your computer probably) to see if the file system has any errors?

9

Show us a minimal reproducible example sketch that demonstrates the problem you say you are encountering.

Tell us what board you're compiling for.

Tell us what libraries you are using.

10

I fixed the issue by adding the line after the SD initialization:

if (SD.exists("/sys/ram.txt")) {SD.remove("/sys/ram.txt"); reset();}

btw reset() is:

void(* reset) (void) = 0;
11

Solved by doing:

myFile = SD.open("/sys/ram.txt", "w");
myFile.close();
myFile = SD.open("/sys/ram.txt", "a+");

More efficient than other method.

12

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.