ESP32 Memory Questions (heap frag related)

Projects Programming
1

Heap Fragmentation Issue on ESP32

Board: Espressif ESP32 Dev Module

Context:

I'm developing a stock control management system using both an offline SQLite3 database (for when there's no connection) and Google Sheets as an online database. The processing is split into two parts:

  1. The ESP32 reads an NFC tag and constructs a GET request to alter or read data from the online database.
  2. Google AppScript modifies the database and returns the results as JSON.

Since I'm aiming to make this device as reliable as possible (with no further modifications after deployment), my main concern is heap fragmentation. I've read several articles about this issue, and I want to minimize it as much as possible. I initially tried using arrays of char, but manipulating them became complex, so I plan to explore other methods first.

Questions:

  1. When I declare a variable globally in Arduino, it is stored in the .data segment of RAM, correct? If I then use reserve to allocate memory (e.g., use varName.reserve(size) in setup()), will it allocate memory from the heap or the data section? I ask this because the answers I found online (including some from ChatGPT) are inconsistent, and I couldn't find many questions directly related to this.
// Global
String payload; // Stored on .bss
String payload2 = "data"; // Stored on .data

// setup
void setup(){
payload.reserve(128); // Is it stored on heap or .bss/.data?
payload2.reserve(128); // Would the old data be deleted?
}
  1. I'm currently using the following approach to reduce heap fragmentation (though I suspect there may be mistakes). Please review and let me know if you see any issues, and suggest improvements if possible!
  • declare null global pointer (to only store the adress of the var in the .data segment);
  • reserve space in setup function (to have static size, i suppose this will alocate space in the heap segment);
  • dereference when needed;
  • never delete it ("static dynamic allocation?").

Now a bit of the code i'm using, i've removed some parts to make it more concise.

Code:

// Global Variables
String *jsonPayload = nullptr;
String *randomStringArray = nullptr;
String *url = nullptr;
String *accessToken = nullptr;
JsonDocument *doc = nullptr;

// Inside setup()
void setup() {
  // Memory allocation
  jsonPayload = new String();
  jsonPayload->reserve(4096);

  accessToken = new String();
  accessToken->reserve(1280);

  url = new String();
  url->reserve(384);

  // Random String Array
  randomStringArray = new String[2];
  randomStringArray[0].reserve(256);
  randomStringArray[1].reserve(256);

  randomStringArray[0] = "1,stock,4984,Video,Camera,Sony,360pMonster,";
  randomStringArray[1] = "1,loan,22,Video,Camera,Sony,360pMonster,22-10-2025 23:59:59";

  // JSON Document
  doc = new DynamicJsonDocument(4096);
}

// Inside HTTP GET function
String httpGET(const String &decodedData) {
  Serial.println("\n\nHTTP Request...\n");

  *url = "";
  *jsonPayload = "";

  *url += BASEURL;
  *url += "?values=";
  *url += urlencode(decodedData);

  Serial.print("\n\nURL ↓\n");
  Serial.println(*url);

  *accessToken = "Bearer ";
  *accessToken += Signer.accessToken();

  http.begin(*url);
  http.setTimeout(50000);
  http.setFollowRedirects(HTTPC_STRICT_FOLLOW_REDIRECTS);
  http.addHeader("Authorization", *accessToken);

  Serial.print("\n\nAuthorization: Bearer ");
  Serial.println(*accessToken);

  int httpResponseCode = http.GET();
  Serial.println("\n\nResponse Code: " + String(httpResponseCode));

  if (httpResponseCode > 0) {
    *jsonPayload = http.getString();
    Serial.println("\n\nResponse ↓\n" + *jsonPayload);
  } else {
    Serial.print("↓ Error: ");
    Serial.print(httpResponseCode);
  }

  http.end();

  return *jsonPayload;
}

This is my first topic here so excuse me for not providing necessary information, any sugestions are welcome! Also, sorry for bad english!

2

The reserved buffer (128 bytes) would be stored on the heap.

In this case, the string object itself will also be stored on the heap.

That being said, I do not think this program would suffer from heap fragmentation in a way that will hurt reliability. If you are worried, I might use the ESP32 heap debugging APIS (Heap Memory Debugging - ESP32 - — ESP-IDF Programming Guide latest documentation) to investigate whether your memory space is becoming more fragmented over time.

1 Like
3

You might also benefit from perusing the esp-idf heap implementation:

It claims relatively low degrees of fragmentation. Presumably it is using slab allocation strategies and other stuff to try to limit how bad things get.

Earlier in my career I worked on a large scale memory cache implementation at a big tech company. We found that total heap fragmentation (then on tcmalloc) for the values we were storing approached about 33% with values in the range of a few bytes to tens of kilobytes. I do not know if the allocator in esp-idf will be as efficient, but as long as the total amount you're allocating is not too large as a fraction of available heap, you probably won't run into trouble.

1 Like
4

Hard to know where to start. You are worried about heap issues but you are using String which is the cause of most heap issues. If you can, use fixed arrays that do not need to be allocated or freed. If you have room left over in Flash progmem, make use of the F macro but only for static strings. Do NOT rely on AI. I am confused about your experience level, you talk about .bss and .data etc like a 10 yr pro, but then you say this is your first post so I wonder if you are in fact a less than 10 yrs experienced person. Heap becomes an issue only if you are frequently doing allocations and freeing. Try to eliminate those but if you can't, schedule a reboot say at midnight or some other idle time.

1 Like
5

I don't think that reserving (enough) memory in setup () would later cause any heap fragmentation. The heap would be allocated at the start and the sketch would keep using already allocated memory without freeing it and allocating it somewhere else which would then leave empty spaces between allocated blocks.

Even if you're using variables, say jsonPayload only for a short time (short-lived Strings and some other objects) in some function the memory would get freed when the function returns.

The only thing that causes memory fragmentation are long lived (global) Strings (and some other objects) that get assigned many times later in the program. In this case, I suppose, reserving enough memory in setup () should do.

You can frequently check if memory gets fragmented by calling heap_caps_get_largest_free_block (MALLOC_CAP_DEFAULT)

2 Likes
6

I moved your topic to a more appropriate forum category @t8h209f0.

The Nano ESP32 category you chose is only used for discussions directly related to the Arduino Nano ESP32 board.

In the future, please take the time to pick the forum category that best suits the subject of your question. There is an "About the _____ category" topic at the top of each category that explains its purpose.

Thanks in advance for your cooperation.

1 Like
7

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.