How to write a uint16_t array into a SPIFFS file

Hi there,

I'm trying to store raw data in the SPIFFS file system, but haven't found the correct way to do so.

For instance, I'm trying to create a file and fill it with the contents of the following const:

const uint16_t imgF[] PROGMEM = {0xF800,0xF840,0xF8A0,0xF900,0xF960,0xF9C0};

I initialise the SPIFFS system, and open a file successfully in the following manner:

File frame0 = SPIFFS.open("/frame0.bin", "wb");

Where I fail is when trying to add content to the file. I've tried in a dozen different manners, to no avail.

For instance:

frame0.write(imgF);

frame0.write(imgF, sizeof(imgF));

for (pixelInc = 0; pixelInc < sizeof(imgF); pixelInc++) { 
   frame0.write(imgF[pixelInc]);
}

I can successfully write txt files with frame0.print("payload"); but not arrays. Would really appreciate the help! :slight_smile:

Post a complete program that we can compile and test on hardware.

Tell us which "Arduino" board you're using.

Forget about the PROGMEM nonsense if you're using an ESP.

Long discussion of using SPIFFS with binary payloads:

https://forum.arduino.cc/t/writing-reading-structs-from-spiffs-on-esp8266/574128

first flaw: if you have data in PROGMEM, you need special measures to access that data.
Read here: http://www.gammon.com.au/progmem to learn how to deal with PROGMEM.

I'm just curious:
Why do you want to keep data in PROGMEM and then copy it to the SD-Card?
What are you going to do with the values (from PROGMEM) written on the SD-Card?

Question #0: what kind of Arduino board?

SPIFFS is often used on esp boards. I'm not sure PROGMEM actually does anything on those boards? Or that reading the data, as though it was in ram, will fail? Von Neumann, not Harvard, so program memory and data memory are in the same address space.

for (pixelInc = 0; pixelInc < sizeof(imgF); pixelInc++) { 
   frame0.write(imgF[pixelInc]);
}

Should be

for (pixelInc = 0; pixelInc < sizeof(imgF)/sizeof(imgF[0]); pixelInc++) { 
   frame0.write(imgF[pixelInc]);
}

What do you mean by "successfully"? What happens when you try to write arrays? In what way is it not successful?

If you are expecting to read the file as a text file somehow, then .print() is the correct function, but .write() will produce random text characters, many of which will not be readable.

Hi gfvalvo,

I'm using an ESF-12F (ESP8266). I can use PROGMEM just fine, such as using arrays to store images that I successfully copy to the display etc. But I can't write it to the Flash via file system, that's all.

Hi PaulRB,

I'm using an ESP-12F (ESP8266). PROGMEM works, I can successfully store images there as on my example above, use it to draw on the screen etc. The only thing I cannot do is to write its data to the file system.

Basically I wanted to move these images to the file system via my code (I know I can upload them via FTP etc.), in order to be able to update them once in a while via code. Unfortunately, with all my examples above, the AVR just crashes hard every time I try to save to the file anything that's not text.

I very much now suspects the SPIFFS documentation is flawed, I have seen posts mentioning mode 'wb' doesn't work (I cannot open files in this mode indeed) despite its documentation stating this is the correct mode for binary files. Its docs also state mode 'w' is for text only, which leaves me with 'w+' only... But I get the same issues with 'w+' that I have with 'w'.

Really at a loss... =/

You still haven't posted a complete code. Do so now, and make it an MRE

Maybe:
https://forum.arduino.cc/t/write-binary-file-in-spiffs-esp8266-32/577007

I would suggest you bookmark:
https://arduino-esp8266.readthedocs.io/en/2.5.0/filesystem.html

Please explain how the AVR is connected to the ESP. Is the PROGMEM data on the AVR and you want to transfer it to the ESP to store in the SPIFFS filesystem?

Right folks, I got insight from mrburnette's post, nailed it. The correct way to deal with the issue is to save one byte at a time, in sequence, rather than trying to save the entire array. The missing trick was a pgm_read_word to ensure I was saving a byte at a time. Here's the code to make it work (followed by the entire code as requested).

Super thanks, folks! Hope this thread helps others in the future. :wink:

  h = 8,w = 8, row, col, buffidx=0;    // Tries to save the image from PROGMEM one pixel at a time. Works!
  for (row=0; row<h; row++) {
    for (col=0; col<w; col++) {
        frame0.write(pgm_read_word(imgF + buffidx));
        buffidx++;
      }
    }

The entire program:

 
#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7789.h> // Hardware-specific library for ST7789
#include "FS.h"              // Flash filesystem SPIFFS
 
// ST7789 TFT module connections
#define TFT_DC   D1     // TFT DC  pin is connected to ESP pin D1 (GPIO5)
#define TFT_RST  D2     // TFT RST pin is connected to ESP pin D2 (GPIO4)
#define TFT_CS   D3     // TFT CS  pin is connected to ESP pin D8 (GPIO0)

// Other pins definitions
const int analogInPin = A0; // Analog in for the three buttons
const int soundPin = D0;    // Audio out is D0 (GPIO16)
int buttonValue = 0;        // Variable that will get the value of the pressed buttons

// Color definitions
#define BLACK    0x0000
#define BLUE     0x001F
#define RED      0xF800
#define GREEN    0x07E0
#define CYAN     0x07FF
#define MAGENTA  0xF81F
#define YELLOW   0xFFE0 
#define WHITE    0xFFFF

// initialize ST7789 TFT library with hardware SPI module
// SCK (CLK) ---> ESP pin D5 (GPIO14)
// MOSI(DIN) ---> ESP pin D7 (GPIO13)
Adafruit_ST7789 tft = Adafruit_ST7789(TFT_CS, TFT_DC, TFT_RST);
 
float p = 3.1415926;

const uint16_t imgF[] PROGMEM = {0xF800,0xF840,0xF8A0,0xF900,0xF960,0xF9C0,0xFA20,0xFA80,0xFAE0,0xFB40,0xFBA0,0xFC00,0xFC60,0xFCC0,0xFD20,0xFD80,0xFDE0,0xFE40,0xFEA0,0xFF00,0xFF60,0xFFC0,0xFFE0,0xEFE0,0xE7E0,0xD7E0,0xCFE0,0xBFE0,0xB7E0,0xA7E0,0x9FE0,0x8FE0,0x87E0,0x77E0,0x6FE0,0x5FE0,0x57E0,0x47E0,0x3FE0,0x2FE0,0x27E0,0x17E0,0xFE0,0x7E0,0x7E1,0x7E3,0x7E4,0x7E6,0x7E7,0x7E9,0x7EA,0x7EC,0x7ED,0x7EF,0x7F0,0x7F2,0x7F3,0x7F5,0x7F6,0x7F8,0x7F9,0x7FB,0x7FC,0x7FE,0x7FF,0x79F,0x73F,0x6DF,0x67F,0x61F,0x5BF,0x55F,0x4FF,0x49F,0x43F,0x3DF,0x37F,0x31F,0x2BF,0x25F,0x1FF,0x19F,0x13F,0xDF,0x7F,0x1F,0x81F,0x101F,0x201F,0x281F,0x381F,0x401F,0x501F,0x581F,0x681F,0x701F,0x801F,0x881F,0x981F,0xA01F,0xB01F,0xB81F,0xC81F,0xD01F,0xE01F,0xE81F,0xF81F,0xF81F,0xF81D,0xF81C,0xF81A,0xF819,0xF817,0xF816,0xF814,0xF813,0xF811,0xF810,0xF80E,0xF80D,0xF80B,0xF80A,0xF808,0xF807,0xF805,0xF804,0xF802,0xF801,
                                 0xF800,0xF840,0xF8A0,0xF900,0xF960,0xF9C0,0xFA20,0xFA80,0xFAE0,0xFB40,0xFBA0,0xFC00,0xFC60,0xFCC0,0xFD20,0xFD80,0xFDE0,0xFE40,0xFEA0,0xFF00,0xFF60,0xFFC0,0xFFE0,0xEFE0,0xE7E0,0xD7E0,0xCFE0,0xBFE0,0xB7E0,0xA7E0,0x9FE0,0x8FE0,0x87E0,0x77E0,0x6FE0,0x5FE0,0x57E0,0x47E0,0x3FE0,0x2FE0,0x27E0,0x17E0,0xFE0,0x7E0,0x7E1,0x7E3,0x7E4,0x7E6,0x7E7,0x7E9,0x7EA,0x7EC,0x7ED,0x7EF,0x7F0,0x7F2,0x7F3,0x7F5,0x7F6,0x7F8,0x7F9,0x7FB,0x7FC,0x7FE,0x7FF,0x79F,0x73F,0x6DF,0x67F,0x61F,0x5BF,0x55F,0x4FF,0x49F,0x43F,0x3DF,0x37F,0x31F,0x2BF,0x25F,0x1FF,0x19F,0x13F,0xDF,0x7F,0x1F,0x81F,0x101F,0x201F,0x281F,0x381F,0x401F,0x501F,0x581F,0x681F,0x701F,0x801F,0x881F,0x981F,0xA01F,0xB01F,0xB81F,0xC81F,0xD01F,0xE01F,0xE81F,0xF81F,0xF81F,0xF81D,0xF81C,0xF81A,0xF819,0xF817,0xF816,0xF814,0xF813,0xF811,0xF810,0xF80E,0xF80D,0xF80B,0xF80A,0xF808,0xF807,0xF805,0xF804,0xF802,0xF801};

 
void setup(void) {
  Serial.begin(9600);
  Serial.print(F("Hello! YouPET initialized."));

  // if the display has CS pin try with SPI_MODE0
  tft.init(240, 240, SPI_MODE0);    // Init ST7789 display 240x240 pixel

  tft.setSPISpeed(64000000);   // Sets SPI speed, default 32000000
 
  // if the screen is flipped, remove this command
  tft.setRotation(3);
 
  Serial.println(F("Initialized"));
  Serial.print('\n');
 
  uint16_t time = millis();
  time = millis() - time;
 
  Serial.println(time, DEC);
  delay(500);

  bool success = SPIFFS.begin();    //  Mounts the filesystem
  if(success){
    Serial.println("File system mounted with success");  
  }else{
    Serial.println("Error mounting the file system");  
  }


}
 
void loop() {

  tft.fillScreen(BLACK);    // Clears screen with black

  tft.fillRect(0, 0, 239, 119, CYAN);
  tft.fillRect(0, 120, 239, 119, RED);
  tft.fillCircle(128, 128, 40, YELLOW);

  File frame0 = SPIFFS.open("/frame0.bin", "w+");
  if (!frame0) {
    Serial.println("file open failed");
  }else{
    Serial.println("file opened");
    Serial.printf("Start Position = %u \n", frame0.position());    // Prints what position in the file we're in
  }

  tft.drawRGBBitmap(30, 30, imgF, 8, 8);   // Draws the image from PROGMEM in its entirety

  int h = 8,w = 8, row, col, buffidx=0;    // Draws the image from PROGMEM one pixel at a time
  for (row=0; row<h; row++) { // For each scanline...
    for (col=0; col<w; col++) { // For each pixel...
        tft.drawPixel(col, row, pgm_read_word(imgF + buffidx));
        buffidx++;
      }
    }

//  frame0.write(imgF);     // Tries to save the contents of imgF to frame0.bin file; doesn't compile.

  h = 8,w = 8, row, col, buffidx=0;    // Tries to save the image from PROGMEM one pixel at a time. Works!
  for (row=0; row<h; row++) {
    for (col=0; col<w; col++) {
        frame0.write(pgm_read_word(imgF + buffidx));
        buffidx++;
      }
    }

  Serial.printf("End Position = %u \n", frame0.position());    // Prints what position in the file we're in. Should be 256.
 
  Serial.println("done");
  delay(5000);
  
}

Only one program? Does that mean there is no AVR involved, just an ESP, and you made a typo before?

If so, ok, but why use PROGMEM at all? Either you are confused, or I am, and if it's me, perhaps the experts here can enlighten me. This is what I think is going on:

PROGMEM is used to store data in the internal flash memory of an AVR, which is useful because many AVR have very limited RAM.

ESP have plenty of flash and RAM, so there's no need to use PROGMEM on ESP. As I said before, and if I'm right, I don't think it actually does anything on ESP. PROGMEM and pgm_read_word have been included in the ESP core just to increase compatibility with code written for AVR chips.

ESP chips don't even have any built-in flash memory. They have external SPI flash chips. Data/code is read from the external flash chip into ram memory and executed from there. On ESP, some ram is used for program execution and some for data, but they have a Von Neumann architecture, so data can be read from any part of the ram memory, and so pgm_read_word is not actually needed. The function has only been included for "backward" compatibility for code written for AVR chips.

This is unlike AVR chips, which have a Harvard architecture, and data cannot be stored or retrieved from program memory (internal flash) except using special instructions, which is what pgm_read_word does.

Can someone please either confirm my theory, or put me straight?

On the ESP8266 PROGMEM is a macro:

#define PROGMEM ICACHE_RODATA_ATTR

ICACHE_RODATA_ATTR is defined by:

#define ICACHE_RODATA_ATTR attribute((section(".irom.text")))

Which places the variable in the .irom.text section in flash. Placing strings in flash requires using any of the methods above.

Declare a global string to be stored in flash.

static const char xyz PROGMEM = "This is a string stored in flash";

https://arduino-esp8266.readthedocs.io/en/latest/PROGMEM.html

I believe you are indeed correct, I’m afraid I’m not familiar enough with these architectures to weight in to that depth.

I’m using PROGMEM to store rather large 16 bits bitmaps and 1 bit sprites that are displayed and animated on a 240x240 SPI display, thus these take a lot of space. It’s not that a have a choice either, as some commands from the GFX library read from PROGMEM such as the one I’ve used in the example above. They can read from SRAM, but although the ESP has more memory (80KB), it’s not like I have enough SRAM to even build a framebuffer of sorts… :slight_smile:

Stored in flash as they may be, they don’t survive a reset or power down, I don’t think. My understanding was that the SPIFFS files do survive a power down, thus why I wanted to store some stuff there as well, persistently.

Hope this makes sense! Again, super thanks for the insights, these have put me way down the progress line of my small game in development.

Paul, one this is for certain: without pgm_read_word I could not write to the file, it wouldn’t even compile in fact. I’ll try drawing to the display without it as you suggest, though — curious about it now!

I'm still confused. As Ray mentioned, PROGMEM does do something, so not as I thought. But if I'm right in saying all flash data (program or data) gets read into ram before use, what is pgm_read_word doing, exactly?

Anyway, I think SPIFFS may be a better place for your bitmaps. But why store them in both PROGMEM and SPIFFS? I seem to remember that you can prepare your SPIFFS files on the PC/laptop, in the same folder, or in a subfolder, of the sketch folder, then they will get uploaded into the SPIFFS flash memory when you upload your sketch. So if you can somehow "compile" your binary files on the PC, maybe you can avoid using PROGMEM.

That I can help clarify. It seems to me that PROGMEM does Not get copied to RAM. I know this for a very simple reason: some of the bitmaps I store in PROGMEM are larger than the 80KB of RAM, thus even the GFX draw command is pulling straight from SPIFFS via PROGMEM.

The reason I'm keeping some bitmaps in PROGMEM is because of the GFX commands. They pull from PROGMEM, there is no way I know off to make drawBitmap to use SPIFFS unless I copy it to RAM first.

Makes sense?

I wonder what you want to express with that sentence.

If you have data in PROGMEM you can read it to RAM.
if you have data in SPIFFS you can read it to RAM.

please make the smallest compileable sketch showing how to deal with your data in PROGMEM.
provide links to all libraries you are using.

and please confirm or revise:
does it mean

  • you have binary data (like an image) currently in PROGMEM which you want to bring ONCE on the filesystem.
  • later on there is no line with PROGMEM any more in your sketch
  • you want to read the previous written image data from filesystem and print the grafic to your display?

There are lots of explanations in this link that are worth reading (if you have not)