[Solved] ESP32 Preferences Storing not working

Hi All,

So, I am using esp32 for my project. I am receiving some data over BLE and I’m trying to store that data in EEPROM of esp32 via the Preferences library. I’m using esp32 BLE UART code to receive data. The data is receiving fine. The problem starts when I’m trying to save the data.

So below is the code -

void ProcessData(char c[], int l){
  char cc[l];
  String s = "";
  for(int i = 0; i < l; i++){
    cc[i] = c[i];
    s = s + cc[i];
    }
    Serial.print("BLE Read Data is: ");
    Serial.println(s);
    Serial.print("BLE Read Data l :");
    Serial.println(s.length());
    if (s == "OTA") {
      Serial.println("OTA enabled for 60 sec");
      StoreData("ble", "false");
      Serial.println(ReadData("ble"));
      delay(500);
      ESP.restart();
    }
}
#include <Preferences.h>
Preferences preferences;

void StoreData(const char* key, const char* val){
  preferences.begin("store",false);
  preferences.putString(key, val);
  preferences.end();
}

String ReadData(const char* val){
  preferences.begin("store",false);
  String ret = preferences.getString(val);
  preferences.end();
  return ret;
}

Now, I have a separate file for each code, BLE.h is where the BLE UART code resides and so is the ProcessData() function and OTA.h is where StoreData() and ReadData() are present, with Preferences library defined and Preferences initiated.

The problem here is - I am receiving data over BLE fine, but the code isn’t storing data to the Preferences.
As seen from the code above, I am storing StoreData(“ble”, “false”); and when I read it back, it still prints “true”!
Since the data is sent from BLE, I can send it multiple times and after few tries it does writes successfully!
Why do you think it’s happening like this?
Help!

PS - I thought it might be because I need to add some delay between read and write to the EEPROM, so I did and it still didn’t work.

Has the Preferences library offer any advantages over using EEPROM, SPIFFS or littleFS for storing persistent data ?

I'm not sure I'm the right guy to answer that. For me, I found preferences was doing what I wanted and moved on, I never explored other options. You think I should? Love to hear more on this.

Can you write a simple piece of test code to store and retrieve a String using preferences?

It's not clear to me from your snippets that you have the syntax correct.

That said, since the esp32 arduino core does have an included eeprom library which supports .put() and .get() functions I have always used it. If you are coming from Arduino land where you are familiar with the eeprom library and .put() and .get() you can use it.

If you are coming from and esp32 world and are familiar using espressif core functions like preferences you can certainly figure out how to get it to work in your context.

A small example of using EEPROM

#include <EEPROM.h>
const byte EEPROM_SIZE = 128;
const byte intSize = sizeof(int);

void setup()
{
  Serial.begin(115200);
  while (!Serial);
  EEPROM.begin(EEPROM_SIZE);
  int rPut;
  int rGet;
  for (int x = 0; x < 8; x++)
  {
    rPut = random(1024);
    Serial.printf("writing %d to EEPROM location %d\n\r", rPut, x * intSize);
    EEPROM.put(x * intSize, rPut);
    EEPROM.commit();
    EEPROM.get(x * intSize, rGet);
    Serial.printf("\tread %d from EEPROM location %d\n\r", rGet, x * intSize);
  }
}

void loop()
{
}

Note that put() and get() support the saving and loading of an array or struct with one command if that is simpler

If you are going to use EEPROM.h you can not directly store and reteive a String object. It has to be converted to the underlying character array (c_string).

The EEPROM is a pseudo EPROM and when you are finished saving your data you need to do a commit(); to actually save it.

when you are finished saving your data you need to do a commit(); to actually save it.

Not when using preferences.h. It's baked into the core library. When using EEPROM.h then you need the commit().

Preferences provides is similar to EEPROM library in Arduino, except that EEPROM provides a single contiguous block of storage which the sketch needs to partition between variables, while Preferences performs the partitioning itself, and therefore it manages the storage locations.

So I Googled a lot after the first reply to consider other storage options and realized that the Preferences actually stores the data in NVS. SPIFFS/LittleFS and EEPROM, for ESP32, are API level different but hardware-level same, with each having its own advantages and disadvantages. For me, EEPROM and Preferences is the correct way since I’m not storing a lot of data.
Also, to clarify, I’m using ESP32-PICO-D4 without external flash/RAM/SD-Card, so everything is inbuilt.

@ cattledog
Sorry for the confusion. In the code posted above the following stores and retrieves the data -

#include <Preferences.h>[color=#222222][/color]
Preferences preferences;

void StoreData(const char* key, const char* val){
  preferences.begin("store",false);
  preferences.putString(key, val);
  preferences.end();
}
String ReadData(const char* val){
  preferences.begin("store",false);
  String ret = preferences.getString(val);
  preferences.end();
  return ret;
}

and they are called from here

// Above is the code to recieve the string via BLE
// It does recieve it successfully, and enters the below if part. 

if (s == "OTA") {
      Serial.println("OTA enabled for 60 sec");
      StoreData("ble", "false");
      Serial.println(ReadData("ble"));
      delay(500);
      ESP.restart();
    }

In serial monitor is can see “OTA enabled for 60 sec” printed, but nothing after that or sometimes “true”
since ReadData(“ble”) is true and to make it false, I have to call StoreData(“ble”, “false”);
Now since the whole If part is called via BLE, I call it multiple times and finally it does write StoreData(“ble”, “false”); successfully.

Also as per your last comment, it looks like preferences and EEPROM are pretty much the same on the API level as well? and if I should change the code to see if the EEPROM library performs better?

@ UKHeliBob
Thanks for the code, I’ll give it a shot later today and will post back here.

@ all
I feel the issue is with threading since the whole storing and reading is called via a BLE trigger and I think on esp32 radios work on a different core than the main thread. I obviously need to Google this. The reason I feel this way is that it works sometimes and doesn’t sometimes and that is the frustrating part, or I may be completely off with this theory.

In serial monitor is can see "OTA enabled for 60 sec" printed, but nothing after that or sometimes "true"

I can't confirm your findings. I am using an ESP32 Dev Module and a BluetoothSerialTerminal on a phone paired and connected to the ESP32.

I have added your StoreData() and ReadData() functions and your "OTA" conditional match to a BLE UART SERVICE sketch which I use for communicating between the phone and the module.

Every time I send "OTA\n" from the phone I see this

OTA enabled for 60 sec
false

@ all
I feel the issue is with threading since the whole storing and reading is called via a BLE trigger and I think on esp32 radios work on a different core than the main thread. I obviously need to Google this. The reason I feel this way is that it works sometimes and doesn't sometimes and that is the frustrating part, or I may be completely off with this theory.

I don't know what to think about your hypothesis, but I can't confirm your finding with a simple BLE message receiving sketch from a phone.

What is sending your BLE message?

I also do not understand how you can be getting "true" read back from the area of storage associated with "ble". You only ever try to store the message "false". Where do you think "true" is coming from?

Thanks for confirming via another esp32 module. I have littlevgl running on esp32 as well and that's why I thought maybe the threading issue.
I'm triggering the ble send from my phone via button and it's fine, since I have been using it for few different apps and devices, all work flawlessly (the ble part I mean).
What I can do is test it on few different pieces of hardware with exact same sketch and will report back.

Regarding where the true is coming from- I check every loop cycle that if esp32 should go to sleep, and if it has to, it stores ble as true before going to sleep, so that everytime it wakes up it initiates ble and connects to my phone. If I manually trigger ble as false, it resets and connects to wifi for OTA update, but only for 60 sec and then sleeps again settings ble as true. So, that was the logic. I have tested this logic and it works fine. To over come the occasional write of ble as false, I added while loop in there so that it doesn't move on till it write ble as false, but now I want to transfer more data from phone to esp32 via ble and while looop does not fit so I need to write to nvm in one go. I will check with few different hardware and report back, as I need this to be fixed.

well, I tested it with different hardware, and I'm facing the same issue. I have to push the data multiple times before it stores it properly. Bummed. :frowning:

I have to push the data multiple times before it stores it properly.

I'm sorry to hear that you are not having success.

My testing with UART SERVICE in a simple sketch reading "OTA" from a terminal program on a phone sent to an ESP 32 with BLE I saw reliable storage.

Have you tried it with EEPROM.h? instead of preferences.h?

You can post more complete code if you want but I don't expect I could see any issues. You are likely the world expert on what you are doing :wink:

So, I did the Core Debug Level: Debug, and got this dump -

[D][BLECharacteristic.cpp:288] handleGATTServerEvent():  - Data: length: 8, data: 7573657220526f79
BLE Read Data is: user Roy
BLE Read Data l :8
Roy
[E][Preferences.cpp:251] putString(): nvs_set_str fail: User INVALID_HANDLE
Read User: [E][Preferences.cpp:444] getString(): nvs_get_str fail: User INVALID_HANDLE

[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLECharacteristic.cpp:285] handleGATTServerEvent():  - Response to write event: New value: handle: 2d, uuid: 6e400002-b5a3-f393-e0a9-e50e24dcca9e
[D][BLECharacteristic.cpp:288] handleGATTServerEvent():  - Data: length: 6, data: 73657878204d
BLE Read Data is: sexx M
BLE Read Data l :6
M
[E][Preferences.cpp:256] putString(): nvs_commit fail: Sex INVALID_HANDLE
Read Sex: [E][Preferences.cpp:444] getString(): nvs_get_str fail: Sex INVALID_HANDLE

[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLECharacteristic.cpp:285] handleGATTServerEvent():  - Response to write event: New value: handle: 2d, uuid: 6e400002-b5a3-f393-e0a9-e50e24dcca9e
[D][BLECharacteristic.cpp:288] handleGATTServerEvent():  - Data: length: 6, data: 687474312035
BLE Read Data is: htt1 5
BLE Read Data l :6
5
[E][Preferences.cpp:251] putString(): nvs_set_str fail: Ht1 INVALID_HANDLE
[E][Preferences.cpp:444] getString(): nvs_get_str fail: Ht1 INVALID_HANDLE

[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLECharacteristic.cpp:285] handleGATTServerEvent():  - Response to write event: New value: handle: 2d, uuid: 6e400002-b5a3-f393-e0a9-e50e24dcca9e
[D][BLECharacteristic.cpp:288] handleGATTServerEvent():  - Data: length: 6, data: 687474322039
BLE Read Data is: htt2 9
BLE Read Data l :6
9
[E][Preferences.cpp:251] putString(): nvs_set_str fail: Ht2 INVALID_HANDLE
[E][Preferences.cpp:444] getString(): nvs_get_str fail: Ht2 INVALID_HANDLE
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLECharacteristic.cpp:285] handleGATTServerEvent():  - Response to write event: New value: handle: 2d, uuid: 6e400002-b5a3-f393-e0a9-e50e24dcca9e
[D][BLECharacteristic.cpp:288] handleGATTServerEvent():  - Data: length: 8, data: 7774747420323435
BLE Read Data is: wttt 245
BLE Read Data l :8
245
[E][Preferences.cpp:251] putString(): nvs_set_str fail: Wt INVALID_HANDLE
[E][Preferences.cpp:444] getString(): nvs_get_str fail: Wt INVALID_HANDLE

So, for example, to understand whats happening, when Im trying to get the user name "User" from nvs, I get this error -
Read User: [E][Preferences.cpp:444] getString(): nvs_get_str fail: User INVALID_HANDLE
Other errors -
[E][Preferences.cpp:256] putString(): nvs_commit fail: Sex INVALID_HANDLE

Read Sex: [E][Preferences.cpp:444] getString(): nvs_get_str fail: Sex INVALID_HANDLE

[E][Preferences.cpp:251] putString(): nvs_set_str fail: Wt INVALID_HANDLE
[E][Preferences.cpp:444] getString(): nvs_get_str fail: Wt INVALID_HANDLE

and so on.
I need to Google why this is happening but it's a start!

Okay, so it is solved.

Problem: In my loop function, I was reading a nvs stored variable that was null, because of which I guess nvs was maybe busy when I was trying to access it via BLE callback.
Solution: I made sure all the stored values in nvs were not null and were read only once at startup or in a particular function when triggered and not in loop function so as to block nvs. So later when BLE was calling nvs store or read functions, it was free and stored/read values nicely.

Thanks for all the help and inputs, especially @cattledog

Cheers!

Well done! Yay :slight_smile: