Storing settings in and syncing them back (ESP32)

Using Arduino Programming Questions
1

Hi, I need help for a project I'm working on.

In this project I have the need of storing some variables in the flash and then read them afterwards.

I'm storing them with a file in littlefs in json format using the ArduinoJson library.

Now the problem is that when I have to sync them, I don't know what kind of data type I'm reading for each combination of keys and values and where I need to sync it to.

So my idea (probably it's dumb but it's what it came in my mind for now) was to create a new struct:

struct SettingDataBase
{
    unsigned int SettingID;
    unsigned int SettingDataType;
    String SettingName;
    String SettingValue;
    String SettingAddresStr;
};

So with that, I can create an array of this new struct as big as I need. This will create my database that I can read with a simple for loop. This struct let me easily add or remove a new voice and, by reading the address of the actual variable, store them directly.

My idea of storing my database (which is simply an array of the new data type "SettingDataBase") was to hardcode it only once by declaring it as a global variable and then, when I need to retrive the setting, sync it back using the database fields.

Now this is where I'm stuck: I don't know if is it possible to cast a String to, for example, an int*, char* and vice versa.

This is complicating my work, maybe there's a simpler and faster way to handle settings stored in the flash which is also easy to maintain in case i need to add or remove elements and that doesn't require to hardcode too much stuff.

Let me know if someone knows a better way to do that.
Thanks and have a nice day.

2

there is no EEPROM in ESP32, so it ends up somewhere in your flash memory

if you want to dump a struct into flash memory, it needs to be flat. the String object holds a pointer to the real text data. so if you archive the String instance, you save the pointer but not the text...

if you really want to store a JSON object, you'll need to serialise it (get a long string and not the object representation) and store that string into memory. it might not be super efficient...

On ESP32 you have the Preferences Library - see how to use it here for example

3

The Preferences library converts String objects to c strings when you use .putString with a String object.

size_t Preferences::putString(const char* key, const String value){
    return putString(key, value.c_str());
}

If you store a struct using the .putBytes() method the Strings in the struct will require conversion.

4

i'm fond of using Type-Len-Values (TLV) for things like this.

there needs to be a table in the code that maps the type to an address in RAM, as well as its size. a multibyte variable or structures can be written as a sequence of bytes following a type and the # of bytes.

a type and length are read, the table is searched for the type, the address located and the values copied to that address. if there is no matching type (e.g. obsolete) the length indicates how many bytes to skip to locate the next type.

a type of 0xFF can mean the last TLV, a value of zero means to ignore it

5

Yes sorry, I wrote EEPROM but what I really meant was that I stored on the flash memory a file in Json format. I've updated the original post.

I thought this method was pretty straight forward but it's not.

Thanks for suggesting this library. Some weeks ago I saw it but I really never investigated that much.

I'll have a look.
Thanks

6

Indeed my comment was not clear enough. I was referring to

I meant if you write the bytes of the Struct in flash (in the same way as if you were writing the bytes of the Struct on a UNO in real EEPROM) then you are toast.

This would fail for example with putBytes in the Preferences Library and you are right, you need to save the members separately using the correct putXXX function depending on the type.

7

Let me repeat so that I know I understood correctly.

Instead of using Strings to store data type, address and value I can use this TLV protocol which will contain all of that.

But this way I have the same problem: how do I link the address of a specific variable allocated as a global variabile to the newly read (from flash mem) value?

Indeed, one of my first attempts was by creating some structures like these:

struct databaseLinker
{
    unsigned int SettingID;
    String SettingName;
};

struct databaseInt
{
    unsigned int SettingID;
    int* SettingAddress;
};

struct databaseFloat
{
    unsigned int SettingID;
    Float* SettingAddress;
};

....and so on....

The SettingName field was used to link the Json doc key and the setting I wanted to sync. The SettingId was used to link this specific settings to the database which contained all the integers addresses for each variable.

But, if I had for example 4 integers, I had to create an array of size 4 elements of the struct "databaseInt". And this story repeats for each data type.

I quickly discarded this method because it was very time consuming to maintain.

8

say your JSON is like this

char json[] = "{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}";

and you have a JsonDocument representing it in your code

DynamicJsonDocument doc(1024);
deserializeJson(doc, json);

the way you could save this in flash memory could be like this

#include <Preferences.h>
Preferences preferences;

// ... later in the code

  // extract the data from the JSON and provide a type
  const char* sensor = doc["sensor"];
  unsigned  time     = doc["time"];
  double latitude    = doc["data"][0];
  double longitude   = doc["data"][1];
  
  // save the data
  preferences.begin("SafeStorage", false);
  preferences. putString("sensor", sensor);
  preferences. putULong("time", time);
  preferences. putDouble("latitude", latitude);
  preferences. putDouble("longitude", longitude);
  preferences.end();

reading that back would be similar, but using getULong() getDouble(). The only catch would be for getString where you would need to provide a pointer to a buffer and not just a pointer without associated storage.

if you insist on saving the JSON in flash, you would have to serialize your doc using serializeJson into a buffer (char array or String) and then use putString()

9

i wasn't suggesting using Strings nor JSON.

but you seem to want to use JSON

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.