Storing data in ESP32

Projects Programming
1

I need to store some data in ESP32. EEPROM is ok, but maybe not enough space.
The other way around is to store it in JSON format, but how? SPIFFS maybe?

2

The ESP32 has no EEPROM. There is a library which uses some of the ESP's Flash memory to emulate EEPROM. You can use as much Flash memory as you like for this, the only limit being the size of the Flash memory. SPIFFS also uses the Flash memory, and so has the same limit.

3

SPIFFS is deprecated in favour of LittleFS
EEPROM is deprecated in favour of, Preferences but you could use either of them depending on the volume of data to be stored

How many bytes of data do you need to store ?

4

I know this. But also, I read that it has a limit of 512 bytes. I will try to expand it.

What about storing JSON data? Say something like this, as is:
{"1":"1","2":"0","3":"0","4":"0"}

JSON could be way bigger. Maybe it could be easier for me to store it somehow this way.

5

JSON is not a type of memory or storage. It is simply a way to format data. You could store data formatted in JSON in either EEPROM or Flash, it would consume the same space. JSON is generally much less efficient than packed binary records, so will use more space.

6

I have read about this, too.

As for data, let's say in numbers.
There should be 4 sets of values. Every set should contain 16 integers from 0-999, and 4 integers from 0-255.

7

That would consume 33 bytes, I think.

Assuming the 4 fields are bytes, only 4 bytes would be needed to store the same 4 fields in packed binary format. If the field are 32-bit ints, then 16 bytes would be needed.

8

{"1":"0","2":"100","3":"120","4":"234","5":"234","6":"1234","7":"1245","8":"124","9":"125","10":"1249","11":"1845","12":"9245","13":"1285","14":"127","15":"1245","16":"100","17":"100","18":"100","19":"0"}

Something like this to be more precise. And four times like this

Also, I made a typo. 16 integers from 0-9999, not 999

9

9999 can be stored as a 16-bit int, which is 2 bytes in binary format. 16 of them would be 32 bytes.

The JSON formatted data will be much longer: each decimal digit and every " , { etc. each takes a byte.

10

EEPROM and Flash have a limited number of writes - how often are you going to be updating the data
consider adding an FRAM
when saving data to EEPROM, FRAM etc avoid Json which is very verbose
save the original binary data and convert to Json when reading it back, e.g. to transmit to the cloud

11

Read/write could go from once a day to some 10 times a day, or so.

This FRAM is interesting...

12

Looks like it has a limit.

And it is 4kB

13

You are correct! Good research.

Have you figured out how much space will be needed, using either packed binary or JSON?

14

ESP32 data storage options

LitteFS

freeRTOS Queue

freeRTOS ring buffer

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html

I like the ring buffer cause it can store stuff and self trigger to send based upon applied parameters and programing. The ring buffer can operate as a background task.

15

Have a look at the FSBrowser example.
It works perfectly well.

16

The problem is that it is not the data that should go to the server. It is the data that is received from the server in JSON format and should be stored in the ESP32 memory for offline work. As there are so many values (some 80 I believe) it will be a pain in the a..s to address them all one by one and to use them. If I could store the whole JSON, no matter the size of the space it will use would be great.

That is why I am here. :slight_smile:

17
1 Like
18

I just did a few tests with this Preferences lib, and it seems it will do the job.

19

Messing around with the Preferences lib I noticed three additional files beside INO: CFG, JSON, and SVD.
Any light here, please?

This is the example I used:


#include <Preferences.h>
#include <WiFi.h>
#include <HTTPClient.h>
Preferences preferences;
WiFiClient client;
String payload;
String data1 = "test";
String data1Out;
void setup() {
  Serial.begin(9600);
  Serial.println(data1);
  preferences.begin("data", false);
  WiFi.mode(WIFI_STA);
  WiFi.begin("ssid", "pass");
  while(WiFi.status() != WL_CONNECTED){
    Serial.print(".");
    delay(100);
    }
  Serial.println();
  preferences.putString("data1", data1);
  data1Out = preferences.getString("data1", "");
  Serial.println(data1Out);
  delay(500);
    if(WiFi.status() == WL_CONNECTED){
    HTTPClient http;
    String server = "http:somesite/index.php";
    http.begin(client, server.c_str());
    int httpCode = http.GET();
    if (httpCode>0) {
    Serial.print("HTTP Response code: ");
    Serial.println(httpCode);
    payload = http.getString();
    Serial.println(payload);
    }
    }
 delay(500);
 preferences.putString("data1", payload);
 Serial.println("End setup");
}

void loop() {
  for(int i=0; i<10; i++){
  data1Out = preferences.getString("data1", "");
  Serial.println(data1Out);
  delay(500);
  }
  delay(500);
  Serial.println("Loop change");
  preferences.putString("data1", data1);
  data1Out = preferences.getString("data1", "");
  Serial.println(data1Out);
  delay(500);
  
}
20

I typically don't hyperventilate about String class use as some here, especially on an ESP32. But why in the world would you create a String object when your only use for it is to access it's internal c-string?