Read/write complex data structure to SPIFFS

Projects Programming
1

Hi. I have an ESP32 project that contains a complex data structure.

The root of the data is a vector array of structures, each of which contain two other vector array of stuctures, each of which….the same.
The length of each part is undefined, as the array length can change, as well as there being String vars which can be different lengths.

At the moment the data is just created by initialisation, but I want the data to be user editable, so I now want to save this data using SPIFFS, read, display, edit and write back.

With the data length being unknown, what is the best method to load and save all this in a byte-wise system? Do I need my own method to store the sizes on write to know the same on read?

Many thanks.
Jim

Here is part of the data structure:

enum modes{alloff, solid, grad, fx};

enum actions{prSolid, prGrad, prFX, BPM, cuRGB, cuHSV, cuGrad, play, stop};

typedef struct
{
  uint8_t hue;
  uint8_t sat;
  uint8_t val;
} CHSV;

typedef struct
{
  uint8_t red;
  uint8_t green;
  uint8_t blue;
} CRGB;

typedef struct
{
  uint hueCount;
  CHSV hsv1, hsv2, hsv3, hsv4;
  uint percentSize;   // 0 for graduated fill, or number of LEDs per step.
  byte direction;
} hsvGrad;

struct action {
  String name;				// Action Name (not used)
  uint8_t type;				// all off, solid, grad, fx or BPM
  uint8_t presetIdx;  // index into slid/grad/fx array
  uint8_t group;			// group to apply action to
  uint8_t BPM;				// BPM for the action (0 leaves tempo unchanged)
  bool    isPlaying;	// is the action playing or static
	unsigned long delay;			// delay (mS) until the next action is triggered
  bool    loop;       // if true, on the last action will loop back to the first action
  CHSV    cHSV;
  CRGB    cRGB;
  hsvGrad cGrad;      // custom Grad
};

struct songPart {
  String  name;
  vector<action> actions;
};

struct song {
    String name;
    vector<songPart> parts;
    vector<songPart> breaks;
};


songPart actionPartOrBreak;

/* All actions for songs go here */
vector<song> songs
{
  {"Song 1",
    {
      songPart{"Intro",
        {
          /*             Type               mS            cHSV     cRGB          cGrad   */
          action{"Start a", cuRGB,0,all,0,false,2000,false, {0,0,0}, {127,0,255},   2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD},
          action{"Start b", cuHSV,0,all,0,false,2000,false, {110,255,255}, {0,0,0}, 2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD},
          action{"Start c",cuGrad,0,all,0,true, 2000,false, {0,0,0}, {0,0,0},       2,{192,255,255},{224,255,255},{0,255,255},{0,255,255},0,FWD},
          action{"Start d",cuGrad,0,all,0,true, 2000,false, {0,0,0}, {0,0,0},       2,{192,255,255},{224,255,255},{0,255,255},{0,255,255},50,BNC}
        }
      },
      songPart{"Part 1",
        {
          action{"Action 1",prSolid,1,all,0,false,1000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}
        }
      },
      songPart{"Part 2",
        {
          action{"Action 1",prSolid,2,all,0,false,1000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}
        }
      }
    },
    {
      songPart{"High Low",
        {
          action{"Action 1",prGrad,0,all,0,true,2000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}
        }
      },
      songPart{"Volume Break",
        {
          action{"Action 1",prGrad,1,all,0,false,1000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}
        }
      },
      songPart{"Stop 1 Bar",
        {
          action{"Action 1",prSolid,4,all,0,false,1000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}
        }
      },
      songPart{"Stop -Tams",
        {
          action{"Action 1",prSolid,0,all,0,false,50,true, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}, //All blue
          action{"Action 1",prGrad,4,tams,0,true,1000,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD}  //Tams CMY
        }
      },
      songPart{"Shoulder",
        {
          action{"Action 1",prGrad,4,all,0,false,250,false, {0,0,0}, {0,0,0},  2,{0,255,255},{0,255,255},{0,255,255},{0,255,255},0,FWD},
        }
      }
    }
  },
2

it's common to stored structured data knowing the size of the structure. a char array of fixed length (e.g.20) can store text data. of course an array of structs can be stored as well

3

Move away from SPIFFS and consider using external storage such as FRAM. As I told another Op, we sometimes forget that freeRTOS is running behind Arduino core on core_1 and core_0 still must handle RF protocols, etc.

SPIFFS performance too bad? - ESP32 Forum

SPIFFS being deprecated · Issue #780 · me-no-dev/ESPAsyncWebServer · GitHub

4

Thanks for your reply, however it isn't the type of storage that is the problem, but how I write/read unknown length data structures into a file.

5

Thanks for the reply.
If I store (say) an array of strings into a byte based file, how do I then extract the strings, when I havent stored the lengths, or the number of items in the array?

6

so by string, we mean c-string, not Arduino/Java String.

a c-string is NUL terminated. so as along as the string < the length of the array - 1, leaving room for the NUL it's end is known. of course this leaves unused byte allocated

7

I am actually using Java String, but I can change it. Thanks.
I guess I will need to write from the array(s) one byte at a time, and put some extra data in there giving array size(s) where necessary.

8

shouldn't be necessary. consider using sprintf() if you need formatted data, if not simple labels

9

I would use ArduinoJson. If you translate your data to JSON format the "Assistant" at ArduionJson.org will generate code to serialize it into a file and deserialize it from a file. For example, your "CHSV" structure could contain three named fields:
{"CHSV" : {"hue": 27, "sat": 120, val:13}}

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.