Storing byte array in SPIFFS

[hutje]

I have a project that uses RAW binary data.
When I currently want to store the binary data, it doesn't seem to go right. The binary data is correct before writing to SPIFFS, after I want to read from SPIFFS and it gives way too many bytes or way too few. And all the values are completely random.
My question is: what is the correct way to write a byte-array to SPIFFS and reading SPIFFS to a byte-array.

The byte array can vary between 60 bytes and 80,000 bytes. The program runs on an ESP32 and has enough RAM to process this size.

[Danois90]

What board are you using? How do you store data to SPIFFS? And how does the code that reads from SPIFFS look like?

[hutje]

I use an ESP32 with 512KB of RAM.
My code for accessing the SPIFFS is fairly simple.

void storeInSPIFFS(byte* payload){
  File storage = SPIFFS.open("/storage.bin", "w+");
  if (storage)
  {
    Serial.println(F("Going to write payload to SPIFFS"));

    //Current size of payload 87 bytes.
//    uint32_t sizePayload = sizeof(payload) / sizeof(uint32_t) - 1;
    size_t sizePayload = sizeof(payload)/sizeof(uint32_t)-1;
//    size_t sizePayload = sizeof(payload); 
    Serial.println(sizePayload); 


    for (int i = 0; i < sizePayload; i++) { //  for debug purposes check the content of the payload
      Serial.print((byte)payload[i]);           //  DEBUG
      Serial.print(":");                        //  DEBUG
    }                                           //  DEBUG
    Serial.println("");                         //  DEBUG



    //WRITE TO SPIFFS
    storage.write(payload, sizePayload); //Also tried fixed value for length of buffer

    //READ FROM SPIFFS TO BYTE-ARRAY
    byte bufferInfo[sizePayload]; //Also tried fixed value
    uint32_t nBytes = storage.readBytes((char*)bufferInfo, sizePayload); //

    for (int i = 0; i < sizePayload; i++) { //Also tried fixed value
      Serial.print((byte)bufferInfo[i]);
      Serial.print(":");
    }
    Serial.print("NBytes:");
    Serial.println(nBytes);
  }
  storage.close();
}
  

[mrburnette]

@hutje

File storage = SPIFFS.open("/storage.bin", "w+");

https://www.esp32.com/viewtopic.php?t=15781

Seems to suggest that the write parameter should be "wb"

Change the fopen mode to “wb”; this will open the file for writing/binary; It will also have the side effect of truncating an existing file, so you can “remove” the remove call. If you want to keep the remove, "wb" or “ab” will do the same with the exception that "ap" will automatically append to an existing file. This is true for the text version as well ("w" vs "a").

Hope there is something there helpful.
The google search was:
https://www.google.com/search?q=SPIFF+Binary+file+site%3Aesp32.com

[Danois90]

There seems to be some issues with the code, look at the comments:

void storeInSPIFFS(byte* payload) { //You should add a parameter with "size in bytes" for the payload
  File storage = SPIFFS.open("/storage.bin", "w+");
  if (storage)
  {
    Serial.println(F("Going to write payload to SPIFFS"));

    size_t sizePayload = sizeof(payload)/sizeof(uint32_t)-1;
    //sizePayload now equals the "number of uint32_t minus one" in payload, why?

    Serial.println(sizePayload); 

    //Here you are printing bytes, not 32 bit integers
    for (int i = 0; i < sizePayload; i++) { //  for debug purposes check the content of the payload
      Serial.print((byte)payload[i]);           //  DEBUG
      Serial.print(":");                        //  DEBUG
    }                                           //  DEBUG
    Serial.println("");                         //  DEBUG

    //Here you are writing a wrong amount of bytes to SPIFFS
    storage.write(payload, sizePayload); //Also tried fixed value for length of buffer

    //Here you are allocating and loading a buffer of wrong size
    byte bufferInfo[sizePayload];
    uint32_t nBytes = storage.readBytes((char*)bufferInfo, sizePayload); //

    //Again, printing wrong data type or size
    for (int i = 0; i < sizePayload; i++) { //Also tried fixed value
      Serial.print((byte)bufferInfo[i]);
      Serial.print(":");
    }
    Serial.print("NBytes:");
    Serial.println(nBytes);
  }
  storage.close();
}
  

[hutje]

Thanks @mrburnette for pointing me to the "wb" parameter!
Thanks @Danois90 for pointing me to the sizes.

I have it working!

void storeInSPIFFS(byte* payload, unsigned int length){
 File storage = SPIFFS.open("/storage.bin", "wb");
 if (storage)
 {
   Serial.println(F("Going to write storage to SPIFFS"));
   Serial.print(F("SizePayload: "));
   Serial.println(length);

   for (int i = 0; i < length; i++) { //  for debug purposes check the content of the payload
     Serial.print((byte)payload[i]);           //  DEBUG
     Serial.print(":");                        //  DEBUG
   }                                           //  DEBUG
   Serial.println("");                         //  DEBUG
   storage.write(payload, length);
 }
 storage.close();

 File readStorage = SPIFFS.open("/storage.bin", "rb");
 if (readStorage) {
   byte buffer[readStorage.size()];
   size_t bufferSize = readStorage.readBytes((char*)buffer,readStorage.size());
   for (int i = 0; i < bufferSize; i++) {
     Serial.print((byte)buffer[i]);
     Serial.print(":");
   }
   Serial.println("");
   readStorage.close();
 }
}