Writing/reading structs from SPIFFS on ESP8266

I am considering using SPIFFS rather than EEPROM (which I use today) to store a configuration struct as a binary data set. I am thinking that the config might eventually exceed 512 bytes (limit of EEPROM).

In EEPROM I simply write or read the struct variable and the size is taken care of by the compiler.

What I wonder regarding SPIFFS is the following:

  1. I write the config struct to a SPIFFS file in binary mode.
  2. The sketch reads the data from SPIFFS at the start and sets up operations.
  3. Down the line I expand the struct size by adding a few new items so it grows in size.
  4. When this sketch runs for the first time it will try to read the struct from a file that does not have enough data yet

What happens? Will the read operation fail and if so how can I find out?

Note that I always add new items to the end of the config struct when it is altered so when reading from EEPROM for the first time I will get bogus data from the following EEPROM locations into these items, but the read succeeds anyway. I have a checksum included at the beginning to see this.
But with a SPIFFS file the size is known to the system so when there are not enough data to read back into the struct, what does it do?
I have read the docs on SPIFFS without getting the wiser.

Read the file and fill in your struct byte-by-byte. When you reach the end of the file (the available() method will return 0), you'll know how many bytes you've read and you know the current size of the struct. If the former is less than the latter, figure out what to do.

To be safe, you should also have a check that you don't over-run the end of your struct.

Will the read operation fail

No. The ESP won't care at all that there is not enough data to fill the struct, or that there is too much data. If there is not enough, what happens depends on where you add fields to the struct.

What happens when there is too much is that you write over memory you don't own, and all kinds of crap can happen.

But with a SPIFFS file the size is known to the system so when there are not enough data to read back into the struct, what does it do?

Whatever you code it to do.

int readSize = file.readBytes((byte*) myConfigStruct, sizeof(myConfigStruct));

will read not more then file size and struct size and returns the size

@Juraj
Perfect! This will work and I can handle the case when the file has not yet been updated with the new items.
I will set the struct to zero before loading from file and then the checksum will still agree if the old data are valid so I can add the last items and save back.

BosseB:
@Juraj
Perfect! This will work and I can handle the case when the file has not yet been updated with the new items.
I will set the struct to zero before loading from file and then the checksum will still agree if the old data are valid so I can add the last items and save back.

one glitch. the readBytes function is blocking. so if the file is shorter then the struct, it will wait for timeout. default timeout is 1 second. you can set file.setTimeout(0).

Juraj:
one glitch. the readBytes function is blocking. so if the file is shorter then the struct, it will wait for timeout. default timeout is 1 second. you can set file.setTimeout(0).

Good catch, thanks for pointing it out. No need for a timeout since there is no other process ever able to write to the file to extend it while I am reading it...

BosseB:
Good catch, thanks for pointing it out. No need for a timeout since there is no other process ever able to write to the file to extend it while I am reading it...

if you add entry to the structure the file will be shorter then the struct so the readBytes will wait for more bytes

I am thinking that the config might eventually exceed 512 bytes (limit of EEPROM).

the maximum is 4096 bytes.

I have now gotten into the coding and was caught by the mode flag...
Apparently the file must be opened in either text or binary mode according to the combined info on:

File system - Arduino core
fopen reference

Is this the case and should I use this to open the file for reading as in this test read function:
(I had to change the cast from (byte*) to (char*) to remove the syntax error in Sloeber...

/***************************************************************************
 * Load configuration data from the SPIFF file, return 0 if OK
 * SPIFFS open function needs path and mode:
 * mode = "r", "w", "a", "r+", "w+", "a+" (text file) or "rb", "wb", "ab", "rb+", "wb+", "ab+" (binary)
 * where r = read, w = write, a = append
 * + means file is open for update (read and write)
 * b means the file os open for binary operations
 * 
 * Returns 0 if OK else:
 * -1 = No SPIFFS file system
 * -2 = File does not exist
 * -3 = File too short
 * -4 = Checksum does not compare
 ***************************************************************************/
int LoadConfigFromFile(ESPConfiguration *Conf, char * configfile)
{
	FSInfo fs_info;
	File F;
	int retcode = 0;
	unsigned int readSize;
	unsigned int chk;
	
	ESPConfiguration Cnf;
	
	SPIFFS.begin();
	
	if (!SPIFFS.info(fs_info)) 
		retcode = -1;	//File system not initialized?
	else
	{
		if (!SPIFFS.exists(configfile)) 
			retcode = -2;	//Could not find config file
		else
		{
			F = SPIFFS.open(configfile, "rb");
			F.setTimeout(0);
			readSize = F.readBytes((char*) Cnf, sizeof(Cnf)); //cast changed from byte*
			if (!(readSize == sizeof(Cnf)))
				retcode = -3;
			else
			{
				chk = Cnf.checksum;
				Cnf.checksum = 0;
				if (CheckSum(&Cnf, sizeof(Cnf)) == chk) //Validated checksum
				{
					Cnf.checksum = chk;
					*Conf = Cnf;
				}
				else
					retcode = -4;
			}
			F.close();	
		}
	}
	SPIFFS.end();
	
	return retcode;
}


/*********************************************************************
 * Write configuration into a SPIFF file at address 0
 * SPIFFS open function needs path and mode:
 * mode = "r", "w", "a", "r+", "w+", "a+" (text file) or "rb", "wb", "ab", "rb+", "wb+", "ab+" (binary)
 * where r = read, w = write, a = append
 * + means file is open for update (read and write)
 * b means the file os open for binary operations
 * 
 * Returns 0 if OK else:
 * -1 = No SPIFFS file system
 * -3 = All bytes could not be written
 *********************************************************************/
int WriteConfigToFile(ESPConfiguration *Conf, char* configfile)
{
	ESPConfiguration Cnf;  //Temp struct
	FSInfo fs_info;
	File F;
	unsigned int writeSize;
	int retcode = 0;
	
	Cnf = *Conf;
	UpdateChecksum(&Cnf);  //Set checksum member to sum of all other bytes

	SPIFFS.begin();
	
	if (!SPIFFS.info(fs_info)) 
		retcode = -1;	//File system not initialized?
	else
	{
		F = SPIFFS.open(configfile, "wb");
		writeSize = F.write((byte*) Cnf, sizeof(Cnf));
		if (!(writeSize == sizeof(Cnf)))
			retcode = -3;
		F.close();	
	}
	SPIFFS.end();
	
	return retcode;
}

Deva_Rishi:
the maximum is 4096 bytes.

I have another problem concerning the EEPROM:
If I need to write different data items into it I have to manage the addresses myself. I failed to do so once when I had miscalculated the size of the struct I used, so there was an overlap.
If I use different file names in SPIFFS instead then I don't have to bother about where the data wind up, right?
So it is a better approach even though the code grows.

BosseB:
I have now gotten into the coding and was caught by the mode flag...
Apparently the file must be opened in either text or binary mode according to the combined info on:

File system - Arduino core
fopen reference

Is this the case and should I use this to open the file for reading as in this test read function:
(I had to change the cast from (byte*) to (char*) to remove the syntax error in Sloeber...

/***************************************************************************
  • Load configuration data from the SPIFF file, return 0 if OK

  • SPIFFS open function needs path and mode:

  • mode = "r", "w", "a", "r+", "w+", "a+" (text file) or "rb", "wb", "ab", "rb+", "wb+", "ab+" (binary)

  • where r = read, w = write, a = append

    • means file is open for update (read and write)
  • b means the file os open for binary operations

  • Returns 0 if OK else:

  • -1 = No SPIFFS file system

  • -2 = File does not exist

  • -3 = File too short

  • -4 = Checksum does not compare
    ***************************************************************************/
    int LoadConfigFromFile(ESPConfiguration *Conf, char * configfile)
    {
    FSInfo fs_info;
    File F;
    int retcode = 0;
    unsigned int readSize;
    unsigned int chk;

    ESPConfiguration Cnf;

    SPIFFS.begin();

    if (!SPIFFS.info(fs_info))
    retcode = -1; //File system not initialized?
    else
    {
    if (!SPIFFS.exists(configfile))
    retcode = -2; //Could not find config file
    else
    {
    F = SPIFFS.open(configfile, "rb");
    F.setTimeout(0);
    readSize = F.readBytes((char*) Cnf, sizeof(Cnf)); //cast changed from byte*
    if (!(readSize == sizeof(Cnf)))
    retcode = -3;
    else
    {
    chk = Cnf.checksum;
    Cnf.checksum = 0;
    if (CheckSum(&Cnf, sizeof(Cnf)) == chk) //Validated checksum
    {
    Cnf.checksum = chk;
    *Conf = Cnf;
    }
    else
    retcode = -4;
    }
    F.close();
    }
    }
    SPIFFS.end();

    return retcode;
    }

/*********************************************************************

  • Write configuration into a SPIFF file at address 0

  • SPIFFS open function needs path and mode:

  • mode = "r", "w", "a", "r+", "w+", "a+" (text file) or "rb", "wb", "ab", "rb+", "wb+", "ab+" (binary)

  • where r = read, w = write, a = append

    • means file is open for update (read and write)
  • b means the file os open for binary operations

  • Returns 0 if OK else:

  • -1 = No SPIFFS file system

  • -3 = All bytes could not be written
    *********************************************************************/
    int WriteConfigToFile(ESPConfiguration Conf, char configfile)
    {
    ESPConfiguration Cnf;  //Temp struct
    FSInfo fs_info;
    File F;
    unsigned int writeSize;
    int retcode = 0;

    Cnf = *Conf;
    UpdateChecksum(&Cnf);  //Set checksum member to sum of all other bytes

    SPIFFS.begin();

    if (!SPIFFS.info(fs_info))
    retcode = -1; //File system not initialized?
    else
    {
    F = SPIFFS.open(configfile, "wb");
    writeSize = F.write((byte*) Cnf, sizeof(Cnf));
    if (!(writeSize == sizeof(Cnf)))
    retcode = -3;
    F.close();
    }
    SPIFFS.end();

    return retcode;
    }

I use only this for compatibility with code that can run with SPIFFS or SD library
#define FILE_WRITE "a"
#define FILE_READ "r"
#define FILE_NEW "w"
I write text and binary files
EDIT: sorry, I checked, I don't write binary data to file on esp8266. I use EEPROM emulation. so I don't know if 'b' is required

Meanwhile I tested the code I posted above and it results in build errors:

..\espconfig.cpp:193:35: error: invalid cast from type 'ESPConfiguration' to type 'char*'
    readSize = F.readBytes((char*) Cnf, sizeof(Cnf));
                                   ^
..\espconfig.cpp: In function 'int WriteConfigToFile(ESPConfiguration*, char*)':
..\espconfig.cpp:283:31: error: invalid cast from type 'ESPConfiguration' to type 'unsigned char*'
   writeSize = F.write((byte*) Cnf, sizeof(Cnf));
                                ^

What is the reason for this?

readSize = F.readBytes((char*) &Cnf, perhaps?

BosseB:
Meanwhile I tested the code I posted above and it results in build errors:

..\espconfig.cpp:193:35: error: invalid cast from type 'ESPConfiguration' to type 'char*'

readSize = F.readBytes((char*) Cnf, sizeof(Cnf));
                                  ^
..\espconfig.cpp: In function 'int WriteConfigToFile(ESPConfiguration*, char*)':
..\espconfig.cpp:283:31: error: invalid cast from type 'ESPConfiguration' to type 'unsigned char*'
  writeSize = F.write((byte*) Cnf, sizeof(Cnf));
                                ^



What is the reason for this?

the function expects a char array, so yes it must by cast to (char*)

Interesting, the F.write() function expects an unsigned char array (AKA byte array) whereas the F.readBytes expects a char array, i.e. a signed byte size item....

One would have expected the opposite, a function readBytes to deal with byte arrays...
So now I use two different casts for the two calls (byte*) and (char*). Then there are no warning/error messages.

Anyway, I have now put some test code into my sketch to check if it works but in fact it failed.
What I did was to add a call to a function InitSPIFFS() in the beginning of setup().
I also added a call there to the function WriteConfigToFile() (shown earlier).

InitSPIFFS() was added specifically since I have not used SPIFFS before and I needed a function that could be called and would check if a format() was needed or not and if so do the formatting.

But I don't even reach the format sentence. So what am I doing wrong? See function below.
Also:

  1. What does SPIFFS.begin() do and can I have a single such call in setup() and then nowhere else?
  2. What does SPIFFS.end() do? Is it needed at all? For example when I send the device to deep sleep do I need this?
    Here is my function that seems to do nothing:
/***************************************************************************
 * Initialize the SPIFFS system if needed
 ***************************************************************************/
void InitSPIFFS()
{
	FSInfo fs_info;

	SerialDebug.println("SPIFFS check T=" + String(millis()));
	SPIFFS.begin();

	if (!SPIFFS.info(fs_info))
	{
		SerialDebug.println("SPIFFS needs formatting T=" + String(millis()));
		if(SPIFFS.format())
		{
			SerialDebug.println("File System formatted T=" + String(millis()));
		}
		else
		{
			SerialDebug.println("File System formatting Error T=" + String(millis()));
		}
	}
	else
	{
		SerialDebug.print("Done! T=" + String(millis()));
		SerialDebug.print(" Size=" + String(fs_info.totalBytes));
		SerialDebug.println(" Used=" + String(fs_info.usedBytes));

		Dir dir = SPIFFS.openDir("/");
		while (dir.next())
		{
			SerialDebug.print(dir.fileName());
		    File f = dir.openFile("r");
		    SerialDebug.println("  " + String(f.size()));
		    f.close();
		}

	}

	SPIFFS.end();
}

My goal is to have a function that can be called on all devices in setup() and if SPIFFS is not initialized it would format SPIFFS.
But from the debug messages on my serial monitor I can see that it does not even reach the format sentence and the write of the config file results in error code -3 (could not write the data)...
Obviously my way of checking if SPIFFS needs formatting is wrong.
Serial monitor display:

SPIFFS check T=5682
Done! T=5689 Size=52961 Used=0
Write conf to file returned: -3

How can one check if SPIFFS is ready for use?

SPIFFS.begin() mounts a valid FS, so don't call it before formatting

http://esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html

Juraj:
SPIFFS.begin() mounts a valid FS, so don't call it before formatting

esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html

According to the document you linked to and I have already read, this is not true.

SPIFFS.begin()

This method mounts SPIFFS file system. It must be called before any other FS APIs are used. Returns true if file system was mounted successfully, false otherwise.

SPIFFS.format()

Formats the file system. May be called either before or after calling begin. Returns true if formatting was successful.

So what is the deal here? It specifically states that format can be done either before or after begin.....

In fact I commented out the SPIFFS.begin and SPIFFS.end commands from my sketch in the format command and tested it. No difference. Nothing shows up...

I performed these steps:

  • Format SPIFFS (this took 790 ms to execute)
  • Write my config struct to a file "/config"
  • Commanded a directory list at "/" (nothing shows up)
  • Read back the file /config to an empty struct, then display its content - still empty

It seems like I am missing something basic here....

Here is the code of my debug command handler for the SPIFFS commands:

				case '8':
				{
							//SPIFF operations:
							// 0= format SPIFF,
							// 1=Read SPIFF directory,
							// 2=Write ESPConf to file /config,
							// 3=Read and show ESPConf
					unsigned long Tstrt, Tend;
					ESPConfiguration Cnf;
					Dir dir1;
					File F;

					/*
					InitSPIFFS();  //Check SPIFFS and format if needed
					strcpy(conffile, "/config");
					int ret = WriteConfigToFile(&ESPConf, conffile);
					SerialDebug.println("Write conf to file returned: " + String(ret));
					*/
					switch (cmd[1])
					{
						case '0':	//format SPIFF
						{
							SerialDebug.println("SPIFFS format");
							//SPIFFS.begin();
							Tstrt = millis();
							SPIFFS.format();
							Tend = millis();
							//SPIFFS.end();
							SerialDebug.println("SPIFFS format took: " + String(Tend - Tstrt));
							strcpy(msg, "SPIFFS formatted");
							ReplyCmd(msg, strlen(msg));
						}
						break;

						case '1':	//Read SPIFF directory
						{
							SerialDebug.println("SPIFFS read dir");
							SPIFFS.begin();
							dir1 = SPIFFS.openDir("/");
							while (dir1.next())
							{
								SerialDebug.print(dir1.fileName());
								F = dir1.openFile("r");
								SerialDebug.println("  " + String(F.size()));
								F.close();
							}
							SPIFFS.end();
							strcpy(msg, "SPIFFS directory");
							ReplyCmd(msg, strlen(msg));
						}
						break;

						case '2':	//Write ESPConf to file /config
						{
							SerialDebug.println("SPIFFS write /config");
							strcpy(msg, "SPIFFS write file /config");
							Cnf = ESPConf;
							WriteConfigToFile(&Cnf, "/config");
							ReplyCmd(msg, strlen(msg));
						}
						break;

						case '3':	//Read and show ESPConf
						{
							SerialDebug.println("SPIFFS read file /config");
							memset (&Cnf, 0, sizeof(Cnf));
							LoadConfigFromFile(&Cnf, "/config");
							PrintConfig(&Cnf);
							strcpy(msg, "SPIFFS /config file read");
							ReplyCmd(msg, strlen(msg));
						}
						break;

						default:
						{
							strcpy(msg, "e8 subcommand not defined");
							ReplyCmd(msg, strlen(msg));
						}
					} //switch
				}
				break;  //subcmd 8

the SPIFFS start address and size is set at build. what is your Flash Size setting in project properties?

(I never formatted SPIFFS because I upload a SPIFFS image with files for web server)

FLASH = 1M(64K SPIFFS)