Problem writing to preferences on ESP32

Projects Programming
1

Hi all!

Trying to save a set of integer values to non-volatile memory in the EPS32, using the preferences library.

Here's my code. When I look at the output, I'm not seeing that the preferences entries are getting stored. The output only seems to include the first and last items. There should be 120 of them, right?

Also, I am seeing the strange line in the serial monitor:

tag = Pg = P9_S8_TUNE

I don't know where this 'Pg' can be coming from!

It seems like either a stupid syntax error on my part, or a strange threading problem?

Thanks in advance!
-eric


#include "Preferences.h"


int currentString = 0;  // tells us which servo to move
int currentPedal = 0;
int currentTune = 500;  // value from 1 to 1000, where 500 is no change for the control.

Preferences copedent;
int pedalCount = 10;
int stringCount = 10;

////////////////////////////////////////// PERSISTENcE  ////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////

const char* getTuneTag(int ped, int str) {
  String tag = "P";
  tag = tag + ped + "_S" + str + "_TUNE"; 
  Serial.println("tag = " + tag);
  const char* result = tag.c_str();
  //if (!copedent.isKey(result)) Serial.println("ERROR: key not found");
  return result;
}

const char* getTag(int IDX, char* type) {
  String tag = "P"; 
  tag += IDX;
  tag += "_";
  tag += type;
  Serial.println("tag = " + tag);
  const char* result = tag.c_str();
  return result;
}

// Storing tuning settings
void writeTuning(int ped, int str, int tune) {            //0-1000
  Serial.print(getTuneTag(ped, str));
  Serial.println(tune);
  copedent.putInt(getTuneTag(ped, str), tune);

}

const char* readTuning(int ped, int str) {
  //Serial.println("reading tuning ");
  String val = "";
  val = val + copedent.getInt(getTuneTag(ped, str));
  Serial.println("val = " + val);
  const char* result = val.c_str();
  
  return result;
}

// Storing the effective range of each control for mapping
void writeMax(int ped, int max) {
  copedent.putInt(getTag(ped, "MAX"), max);
  //Serial.println(getTag(ped, "MAX"));
}

void writeMin(int ped, int min) {
  copedent.putInt(getTag(ped, "MIN"), min); 
  //Serial.println(getTag(ped, "MIN"));
}

int readMax(int ped) {
  copedent.getInt(getTag(ped, "MAX"));
}

int readMin(int ped) {
  copedent.getInt(getTag(ped, "MIN"));
}

void createDefaultPrefs() {
  Serial.println("Creating Prefs");
  for (int ped = 0; ped < pedalCount; ped++) {
    writeMax(ped, 14400);
    writeMin(ped, 2000);
    for (int str = 0; str < stringCount; str++) {
      writeTuning(ped, str, 500);
    }
  } 
}


void setup() {
  Serial.begin(115200);
  while (!Serial) {
    delay(10);
  }

  copedent.begin("tuning", false);
  bool notNew = copedent.isKey("P0_S0_TUNE");
  if (notNew == false) {
    createDefaultPrefs();
  }
} 

void loop() {
  // put your main code here, to run repeatedly:

}
2

I left my computer for the day, and came back to see the entire correct list of tags in my serial printout!

Wondering if I had changed something the last time I pasted into this forum, I sent the sketch to the esp again. I did not see the entire list. Instead, I saw something that has happened many times before. Partial lines showing up in the serial monitor. It suggests to me I'm sending data out too fast, but I really don't know. I saw the option to toggle timestamp, and everything in the monitor, except the first entry, has the same timestamp. Nothing else got printed:

1:35:32.381 -> ag = P9_S8_TUNE
21:35:32.381 -> tag = P9_Build:Mar 27 2021
21:35:32.381 -> rst:0x1 (POWERON),boot:0x28 (SPI_FAST_FLASH_BOOT)
21:35:32.381 -> SPIWP:0xee
21:35:32.381 -> mode:DIO, clock div:1
21:35:32.381 -> load:0x3fce2820,len:0x10cc
21:35:32.381 -> load:0x403c8700,len:0xc2c
21:35:32.381 -> load:0x403cb700,len:0x30b0
21:35:32.381 -> entry 0x403c88b8
21:35:32.381 -> Creating Prefs
21:35:32.381 -> tag = P0_MAX
21:35:32.381 -> tag = P0_MIN
21:35:32.381 -> tag = ag = P9_S8_TUNE
21:35:32.381 -> tag = P9_S9_TUNE
21:35:32.381 -> 500
21:35:32.381 -> tag = P9_S9_TUNE

Is there something that needs to be initialized differently in the setup of the serial monitor?

In any case, the data is not actually getting written or else isn't being read properly, or my test tag would return a value, and 'create preferences' wouldn't run....

3

Okay, I went back to the original code I got this from, and realized I didn't get the whole process into my code. It looks like I need to call preferences.end, and restart the eps. I'll try that tomorrow....

4

That might be due to buffering of received data by the PC.

You can add something like Serial.println("Prefences test"); in setup() before printing anything else. Anything that is printed before that is old stuff.

5

Both functions build a local String called tag, then return tag.c_str(), which is a pointer into that String's internal buffer.

But as soon as the function returns, the String object is destroyed and its buffer can be freed or reused by the next String constructed anywhere in the program. The const char * you get back is a dangling pointer, and what it points to depends on whatever memory gets reused next...

Right after the String is destroyed, its old buffer content may still be sitting in freed heap memory untouched, so the c_str() pointer looks valid until something else allocates over it. Since your code immediately calls another function that also constructs and destroys a String, you get overlap and corruption most of the time

➜ it likely explains the kind of intermittent, order-dependent corruption you are seeing, including the mysterious "Pg" and the truncated printouts.

if you want to use c-string, make sure you have a permanent buffer (provided by the caller ideally) or as you are using String anyway, return a copy of the String.

PS/ I have not looked at your code in details but NVS keys in ESP32 Preferences have a 15 character limit. You seem to be building that dynamically, so ensure it stays within limits.

6

Thanks so much for that!

So I should just return the tag String? The methods of preferences take a *char array. Can I return the String and convert it in the get or put call? Like so:

String getTuneTag(int ped, int str) {
  String tag = "P";
  tag = tag + ped + "_S" + str + "_TUNE"; 
  Serial.println("tag = " + tag);
  //const char* result = tag.c_str();
  //if (!copedent.isKey(result)) Serial.println("ERROR: key not found");
  return tag;
}

String readTuning(int ped, int str) {
  //Serial.println("reading tuning ");
  String val = "";
  val = val + copedent.getInt(getTuneTag(ped, str).c_str());
  Serial.println("val = " + val);

Or is there still a risk of it being overwritten? That's what I'll try, I guess.

As a Java programmer, both pointers and strings in c++ are very confusing to me. I'm not the least surprised I messed this up.

I did notice the character limitation, my maximum key length is 11 characters.

7

Why use a String in the first place rather than an array of chars built using the snprintf() function ?

8

Okay, it seems that c_str() still points to invalid memory.

What is the suggested method to convert my String to a pointer to a character array?

I'll try to figure that out.

Why use a String in the first place rather than an array of chars built using the snprintf() function ?

Because one has never heard of such a function? ;)

I'll look into that!

9

Yes that’s a possibility. What it does is silently duplicate the string you had locally and hand it over to the caller

String tuneTag =  getTuneTag(3,4); // whatever

Then if you need the underlying buffer, use tuneTag.c_str() indeed

(But that’s a lot of memory wasted - on an esp32 you have more than on a small uno so it might be fine but you need to be aware of the underlying cost).

10

I tried this:

char* getTuneTag(int ped, int str) {
  char tag[11];  // max length you’ll need +1
  snprintf(tag, "P%d_S%d_TUNE", ped, str);
  Serial.println("tag = " + tag);
  return tag;
}

The examples show using integers in the snprintf function, but this gives a compiler error. Int can't convert to string.

from the reference:

  cx = snprintf ( buffer, 100, "The half of %d is %d", 60, 60/2 );

Why doesn't it like my integers???

EDIT: Okay now I see. I need to declare the size again. That compiles, but I guess the + operator can't be used with char*? What's the method I should be using to concatenate the strings?

I'm also not sure I've fixed the issue with the string getting overwritten. Does my buffer need to exist outside the function??

11

Okay, I am trying to continue to simplify my code to track down what's happening.

I tried this:

void writeTuning(int ped, int str, int tune) {            //0-1000  
  char tag[11];
  copedent.putInt(snprintf(tag, 11, "P%d_S%d_TUNE", ped, str), tune);
}

And I get this error:

error: invalid conversion from 'int' to 'const char*' [-fpermissive]
   37 |   copedent.putInt(snprintf(tag, 11, "P%d_S%d_TUNE", ped, str), tune);
      |                   ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                           |
      |                           int

Why does it think the first argument should be an int? Or is an int? Neither is true, is it?

This from prefs docs:

putInt(const char* key, int32_t value)
12

Same issue as before

Tag is a local variable in the function so when you return the pointer and the function terminates the storage is returned to global stack use and will be overwritten

It’s time to read about scope :slight_smile:

13

yeah, I see the scope issue.

But I did fix that in the subsequent version, correct? Where the tag is within the same scope as the call to putInt?

This compiles:

void writeTuning(int ped, int str, int tune) {            //0-1000  
  char tag[11];
  snprintf(tag, 11, "P%d_S%d_TUNE", ped, str);
  copedent.putInt(tag, tune);
}

Oh, yeah, I remember now. The function returns an int of its length..... Maybe I can get there from here ;)

14

:slight_smile: you are on the right track

(To be exact snprintf returns the number of characters that would have been written if the buffer had been large enough, not counting the terminating null byte.)

15 
snprintf(tag, 11, "P%d_S%d_TUNE", ped, str);

would be safer as

snprintf(tag, sizeof(tag), "P%d_S%d_TUNE", ped, str);
16

It's very common to add an integer to a pointer. For example, if the C-string starts at address 0x1000, then adding 12 to point at 0x100c is the 13th char of that string (if it is at least that long). But adding two pointers will almost always point to same random location in memory: 0x1000 + 0x8800 = 0x9800; what's there? Concatenating C-strings is a more complicated operation; you must allocate the resulting memory first, and that doesn't happen automatically like it does in Java.

Arduino added the String class for all these kinds of conveniences (and all the methods that JavaScript also inherited from Java, like lastIndexOf). But they could not add a compacting garbage collector. So if you use it, you have to be careful not to fragment the heap so much that your firmware will eventually fail.

That is basically unrelated to the scope issues you have been dealing with. In Java, all objects are allocated on the heap, so they persist when returning from a function. Not so with C++, which does stack allocation of local variables in a function. One quick improvement would be to declare it like this

  static char tag[11];

so that it is no longer strictly local, and is reused for every function invocation. It also means that when you return tag, that pointer is valid outside the function.

However, your current track, calling the Preferences function from within a helper function using a local pointer, is the safer way to go.

17

True but not a great practice so should not be recommended

18

Very informative, thanks! I am seeing that there is a lot I've been able to take for granted, working with Java.