Where goes the (ESP32-) Memory?

Projects Programming
1

I do have a rather large project on ESP32s, and I know I consume plenty of memory, yet while I seem to have enough memory left, I can't increase a buffer for only a fraction of the available memory.

Is there something else consuming memory, which isn't measured by these ESP functions? I don't understand. Help urgently needed.

Using the ESP functions:

ESP.getFreeHeap(), ESP.getMinFreeHeap(), ESP.getHeapSize(), ESP.getMaxAllocHeap()

and a buffer defined globally as:

char BufOther[4500];

I get this output:

Heap: Free:67760, Min:50436, Size:287764, Alloc:49260

This looks to me like I could allocate some 49kB and would be ok. My buffer is supposed to hold a dynamic web page, for which I need at least double its current size.

Yet, when I increase the buffer size by only 500 Bytes, the program crashes right after starting.

(Not a stack issue: the stack high water mark of the 'loop' is ~5000, and the buffer is defined globally anyway)

What am I missing?

2

Please post all the code as an attachment.

3

I am afraid, posting all the code is neither legally possible, nor awfully meaningful - the full code is proprietary, and the whole thing is 1 MB.

What puzzles me is that there is much free heap, yet I can't increase a buffer by a tiny fraction of that heap?

I wondered whether that heap is really there, and used malloc() to create a new buffer. It succeeded up to and including 49260 bytes - the number given by ESP.getMaxAllocHeap() (see initial posting). Asking for 1 byte more and it fails.

So, the heap is available! Why can I not use it for a buffer defined globally (i.e. before setup())?

There has got to be something else which is limiting what kind of memory is taken by whatsoever?

4

The thing is, you are focusing on the system rather than the more likely cause, which is a programming error that you have made. Think about the large number of programs that are out there, and work quite well with dynamic memory. If there was a system problem, it would have been reported by hundreds or thousands of users already. It's unfortunate that you can't post the code, although it's unlikely that anyone here would be willing to sift through 1Mb. I'm sure you have already discovered in the past, that memory overflows and buffer miscalculations lead to situations where the symptoms don't match the cause, or provide much insight into the cause. That is why we almost always ask for complete code samples here.

5

Given the information that helps me help you, I'd say the issue to the problem can be found in the ESP32 API, API Reference - ESP32 - — ESP-IDF Programming Guide latest documentation

6

@aarg: I am totally with you, but any straight forward procedure is not available here. I had hoped that my description would ring a bell in someone's mind and he'd present an idea. Something ought to be different beyond simple heap and stack.

@Idahowalker: that link goes to an Everest of information, and even if I were going though it, I could well miss the relevant information. Nevertheless, I tried just that, and came across Heap Memory Debugging:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/heap_debug.html#heap-information

They talk about multiple heap types being available on ESP32s, and functions to sample various details. It sounds relevant to my case.

But if I am not mistaken, those functions are not available under Arduino? At least I don't know more than the 4 heap related ones I have used. Anyone knows more?

I'll try if a simple demo could demonstrate the problem

7

SO far, I have went though a bit more then 70% of the functions listed in the ESP32 API, so far every one of them has worked with the Arduino API.

8

So, here is the demo. I have shrunk my code from 1MB to under 1k :frowning:

// ullix's heap problems

#define bbsize 109100
byte BigBuffer[bbsize] = {0};
uint32_t LastLoopStart;

String getAllHeap(){
char temp[300];
sprintf(temp, "Heap: Free:%i, Min:%i, Size:%i, Alloc:%i", ESP.getFreeHeap(), ESP.getMinFreeHeap(), ESP.getHeapSize(), ESP.getMaxAllocHeap());
return temp;
}

void setup(){
Serial.begin(115200);
Serial.setDebugOutput(true);

for (uint32_t i =0; i< bbsize; i++) BigBuffer = 0x65;
}
void loop(){

  • if ((micros() - LastLoopStart) >= 1000000ul){*
  • Serial.print(getAllHeap());*
  • Serial.printf(", Stack HWM: %i \n", uxTaskGetStackHighWaterMark(NULL));*
  • LastLoopStart = micros();*
  • } *
    }
    [/quote]
    I am running the UECIDE IDE, but I have verified that the code behaves in the same way on Arduino IDE.
    When I run this code, the compiler tells me this memory usage:
    > Memory usage
    > • Program size: 220.948 bytes
    > • Memory size: 170.740 bytes
    > • Compilation took 1,442 seconds.
    The output to serial is a steady:
    > Heap: Free:253092, Min:246600, Size:279052, Alloc:113792, Stack HWM: 6440
    > Heap: Free:253092, Min:246600, Size:279052, Alloc:113792, Stack HWM: 6440
    > Heap: Free:253092, Min:246600, Size:279052, Alloc:113792, Stack HWM: 6440
    (Actually, after some 10 min the Stack HW has changed to 6392, while heap remained unchanged)
    Tell me if I am wrong, but i read this has having the option to allocate >113 KILO-BYTES?
    Now increase the var bbsize from 109100 to 109200, increasing the buffer by only 100 BYTES. The compiler now tells me:
    > Compiling...
    > • Preprocessing...
    > • Converting binary files
    > • Compiling sketch...
    > ◦ test_heapee_combined.cpp
    > • Compiling core...
    > ‣ base64
    > ‣ esp32
    > • Compiling libraries...
    > • Linking sketch...
    > /home/ullix/.uecide/compilers/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: /home/ullix/Code/uecide/test_heapee/build/test_heapee.elf section .dram0.bss' will not fit in region dram0_0_seg'
    > /home/ullix/.uecide/compilers/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: DRAM segment data does not fit.
    > /home/ullix/.uecide/compilers/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: region `dram0_0_seg' overflowed by 88 bytes
    > collect2: error: ld returned 1 exit status
    > Compiling Failed
    Overflowed by 88 bytes? what is it that overflows, and what function tells me that?
9

What happens if you declare two buffers?

#define bbsize 109100
byte BigBuffer1[bbsize] = {0};
byte BigBuffer2[bbsize] = {0};

My guess is that there is a limit to the size of static allocations. You could try to allocate the buffers dynamically with "malloc()" and see how it goes.

10

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/system.html?highlight=heap%20free%20size

I do not beleive the result set is returning in Kb.

esp_get_free_heap_size() returns the current size of free heap memory

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/mem_alloc.html

You might want to look to see which memory partition scheme works for you.
xPortGetFreeHeapSize() is a FreeRTOS function which returns the number of free bytes in the (data memory) heap.

Many of the Arduino thingies will cause freeRTOS to load and it might be a issue source?

I'd not use millis() with an ESP32. Instead use esp_timer_get_time(void) which will roll over in100 plus years; see High Resolution Timer (ESP Timer) - ESP32 - — ESP-IDF Programming Guide latest documentation

11

Neither global variables not local variables are allocated on "the heap." It could be that the ESP environment reserves a certain amount of memory for the heap, and therefore does not have as much RAM space left for non-dynamic storage.

12

westfw:
Neither global variables not local variables are allocated on "the heap." It could be that the ESP environment reserves a certain amount of memory for the heap, and therefore does not have as much RAM space left for non-dynamic storage.

It "could", sure. But where exactly does it go, which function tells me, how much space is left, and how can I increase this space?

No, the global and local vars should not be taken from the heap, but my understanding is that the heap should be what is left after all vars have taken their space?

With a FreeHeap of some 250kB my DRAM has plenty of empty space!

13

Unsurprisingly, I am not the first one who got puzzled by the issue presented here. The clue is in this statement:

from Heap Memory Allocation - ESP32 - — ESP-IDF Programming Guide latest documentation

"Note:
Due to a technical limitation, the maximum statically allocated DRAM usage is 160KB. The remaining 160KB (for a total of 320KB of DRAM) can only be allocated at runtime as heap."

It is discussed here https://github.com/espressif/esp-idf/issues/3497 and more here maximum ram usable is 96000 · Issue #1163 · espressif/arduino-esp32 · GitHub

Overall, while the ESP32s have much RAM, it is not necessarily freely usable. For global vars the limit seems to be somewhere near 100kB. This makes the ESP32 much less useful than it seemed.

Heap can be used only by malloc().

I still don't know any function which lets me determine what RAM I have left for global vars. Any help is much appreciated.

14

I'd just try and malloc the buffer you need in setup. It's not as though you'll ever free it, so it should not cause fragmentation issues.

15

yes, why not use that wasteland-memory with everything that can be malloc'ed :smiling_imp:

16

It is discussed here https://github.com/espressif/esp-idf/issues/3497 and more here maximum ram usable is 96000 · Issue #1163 · espressif/arduino-esp32 · GitHub

Overall, while the ESP32s have much RAM, it is not necessarily freely usable. For global vars the limit seems to be somewhere near 100kB. This makes the ESP32 much less useful than it seemed.

First of all, thank you for tracking down those references and reporting back here...

Heap can be used only by malloc().

Note that your particular problem from the original post can be "adequately" solved my using malloc() to get your buffer, rather than statically allocating it. (although there are frequently reasons to avoid malloc(), it seems like this situation would warrant an exception!)

`dram0_0_seg' overflowed by 88 bytes
Overflowed by 88 bytes? what is it that overflows, and what function tells me that?

dram0_0_seg overflowed, just like it says in the error message. Since it is happening at link time, there isn't a function to tell you about it, just the linker error message.

    Global variables use 122604 bytes (37%) of dynamic memory, leaving 205076 bytes for local variables. Maximum is 327680 bytes.

This "normal compile" message seems very misleading, if in fact the limit for the memory it's talking about is only about 120k, rather than the ~300k theoretically available on the chip. Might it be worth submitting a bug report...

Now, since my first guess was right, I might as well keep on guessing! I don't know what "technical issue" prevents the ESP32 from having all of the RAM accessible to the compiler. Sometimes RAM in a chip is not contiguous (one block at 0x10000000, another at 0x2000 0000, neither 16MB long), and this can make things more difficult for a compiler, and prevent single data elements from being longer than a memory segment, but I thought that gcc would handle that sort of thing better than it is doing here.

I still don't know any function which lets me determine what RAM I have left for global vars. Any help is much appreciated.

dram0_0_seg will be a variable defined by the tools in something called a "linker script" that is invoked by the link command issued by the IDE with the -T switch. In this case:

.../packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-gcc" -nostdlib "-L/Applications/Arduino-1.8.9 copy.app/Contents/Java/portable/packages/esp32/hardware/esp32/1.0.2/tools/sdk/lib" "-L/Applications/Arduino-1.8.9 copy.app/Contents/Java/portable/packages/esp32/hardware/esp32/1.0.2/tools/sdk/ld" [color=teal][b]-T esp32_out.ld[/b][/color] -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.spiram_incompatible_fns.ld -u ld_include_panic_highint_hdl ...

(Oh wow - there are multiple -T switches. I didn't even know you could do that!)
A linker script describes the memory layout of a physical chip, and how the "logical" sections that the compiler generates should be fit into that memory.

The esp32_out.ld is the linker script that describes the RAM. It's pretty obscure stuff, but there are comments. It says (among other things):

  /* Shared data RAM, excluding memory reserved for ROM bss/data/stack.
     Enabling Bluetooth & Trace Memory features in menuconfig will decrease
     the amount of RAM available.

     Note: Length of this section *should* be 0x50000, and this extra DRAM is available
     in heap at runtime. However due to static ROM memory usage at this 176KB mark, the
     additional static memory temporarily cannot be used.
  */
  dram0_0_seg (RW) : org = 0x3FFB0000 + 0xdb5c,
                                     len = 0x2c200 - 0xdb5c

I guess ... The ROM functions use a chunk of the RAM in the middle? (probably left over from earlier chips with less total RAM? malloc() and etc may get info about the actual RAM usage by the ROM at runtime, which could be what permits it to be used at all.)
THAT "len" (124580 bytes) seems to match up with the limits you are seeing.

There are some additional tools that will help you look at memory usage of your sketch, they live off in the ESP32 tools directory ( ...packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/ )
(You'll also need to find the actual location where the IDE has put the sketch binaries.)

size (xtensa-esp32-elf-size) - reports size of the various sections:
.../packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-size /tmp/NewArduBuild/test_esp32_heap.ino.elf 
   text    data     bss     dec     hex filename
 150840   51428  114040  316308   4d394 /tmp/NewArduBuild/test_esp32_heap.ino.elf

nm (xtensa-esp32-elf-nm) - reports on symbols in the file, especially useful in the form:

xtensa-esp32-elf-nm -SC --size-sort mysketch.elf
4008a0ac 00000490 T rtc_init
400d7400 00000498 T nvs::Page::mLoadEntryTable()
400d7af0 00000544 T nvs::PageManager::load(unsigned int, unsigned int)
3ffdae24 00000800 b s_stub_min_stack
3f401810 00000820 D rtc_gpio_desc
400e1df4 00000ba2 T _dtoa_r
400e3004 00001ffa T _svfiprintf_r
400dc338 000020fe T _vfiprintf_r
400d8d6c 000030de T _svfprintf_r
400de970 000031e2 T _vfprintf_r
3ffbfcf0 0001aa2c B BigBuffer

(The "T", "B", "D" labels indicate whether the object is in text (code), BSS, or Data sections.)

17

It is discussed here https://github.com/espressif/esp-idf/issues/3497 and more here maximum ram usable is 96000 · Issue #1163 · espressif/arduino-esp32 · GitHub

Overall, while the ESP32s have much RAM, it is not necessarily freely usable. For global vars the limit seems to be somewhere near 100kB. This makes the ESP32 much less useful than it seemed.

First of all, thank you for tracking down those references and reporting back here...

Heap can be used only by malloc().

Note that your particular problem from the original post can be "adequately" solved my using malloc() to get your buffer, rather than statically allocating it. (although there are frequently reasons to avoid malloc(), it seems like this situation would warrant an exception!)

`dram0_0_seg' overflowed by 88 bytes
Overflowed by 88 bytes? what is it that overflows, and what function tells me that?

dram0_0_seg overflowed, just like it says in the error message. Since it is happening at link time, there isn't a function to tell you about it, just the linker error message.

    Global variables use 122604 bytes (37%) of dynamic memory, leaving 205076 bytes for local variables. Maximum is 327680 bytes.

This "normal compile" message seems very misleading, if in fact the limit for the memory it's talking about is only about 120k, rather than the ~300k theoretically available on the chip. Might it be worth submitting a bug report...

Now, since my first guess was right, I might as well keep on guessing! I don't know what "technical issue" prevents the ESP32 from having all of the RAM accessible to the compiler. Sometimes RAM in a chip is not contiguous (one block at 0x10000000, another at 0x2000 0000, neither 16MB long), and this can make things more difficult for a compiler, and prevent single data elements from being longer than a memory segment, but I thought that gcc would handle that sort of thing better than it is doing here.

I still don't know any function which lets me determine what RAM I have left for global vars. Any help is much appreciated.

dram0_0_seg will be a variable defined by the tools in something called a "linker script" that is invoked by the link command issued by the IDE with the -T switch. In this case:

.../packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-gcc" -nostdlib "-L/Applications/Arduino-1.8.9 copy.app/Contents/Java/portable/packages/esp32/hardware/esp32/1.0.2/tools/sdk/lib" "-L/Applications/Arduino-1.8.9 copy.app/Contents/Java/portable/packages/esp32/hardware/esp32/1.0.2/tools/sdk/ld" [color=teal][b]-T esp32_out.ld[/b][/color] -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.spiram_incompatible_fns.ld -u ld_include_panic_highint_hdl ...

(Oh wow - there are multiple -T switches. I didn't even know you could do that!)
A linker script describes the memory layout of a physical chip, and how the "logical" sections that the compiler generates should be fit into that memory.

The esp32_out.ld is the linker script that describes the RAM. It's pretty obscure stuff, but there are comments. It says (among other things):

  /* Shared data RAM, excluding memory reserved for ROM bss/data/stack.
     Enabling Bluetooth & Trace Memory features in menuconfig will decrease
     the amount of RAM available.

     Note: Length of this section *should* be 0x50000, and this extra DRAM is available
     in heap at runtime. However due to static ROM memory usage at this 176KB mark, the
     additional static memory temporarily cannot be used.
  */
  dram0_0_seg (RW) : org = 0x3FFB0000 + 0xdb5c,
                                     len = 0x2c200 - 0xdb5c

I guess ... The ROM functions use a chunk of the RAM in the middle? (probably left over from earlier chips with less total RAM? malloc() and etc may get info about the actual RAM usage by the ROM at runtime, which could be what permits it to be used at all.)
THAT "len" (124580 bytes) seems to match up with the limits you are seeing.

There are some additional tools that will help you look at memory usage of your sketch, they live off in the ESP32 tools directory ( ...packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/ )
(You'll also need to find the actual location where the IDE has put the sketch binaries.)

size (xtensa-esp32-elf-size) - reports size of the various sections:

.../packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-size /tmp/NewArduBuild/test_esp32_heap.ino.elf 
   text    data     bss     dec     hex filename
 150840   51428  114040  316308   4d394 /tmp/NewArduBuild/test_esp32_heap.ino.elf

nm (xtensa-esp32-elf-nm) - reports on symbols in the file, especially useful in the form:

xtensa-esp32-elf-nm -SC --size-sort mysketch.elf
4008a0ac 00000490 T rtc_init
400d7400 00000498 T nvs::Page::mLoadEntryTable()
400d7af0 00000544 T nvs::PageManager::load(unsigned int, unsigned int)
3ffdae24 00000800 b s_stub_min_stack
3f401810 00000820 D rtc_gpio_desc
400e1df4 00000ba2 T _dtoa_r
400e3004 00001ffa T _svfiprintf_r
400dc338 000020fe T _vfiprintf_r
400d8d6c 000030de T _svfprintf_r
400de970 000031e2 T _vfprintf_r
3ffbfcf0 0001aa2c B BigBuffer

(The "T", "B", "D" labels indicate whether the object is in text (code), BSS, or Data sections.)

18

Let me first clarify one thing, which keeps popping up here: my problem were NOT the big buffers! I simply put them in for the sole purpose of gobbling up some memory to demonstrate the problem in a small demo file.

My real problem was the creation of a dynamic website, using Strings (capital S) to assemble the HTML code. I couldn't figure out what the bugs were in my code that resulted in garbled web pages - text missing in the beginning, in the end, in the middle, or garbage-only everywhere.

Tried to use string (small cap s) but it was more complicated and it also failed.

No surprise that I couldn't find a bug in my code, because there was none! The real issue became clear only in the course of this posting, so at least it was of great help!

I was a bit generous in my memory usage, because the heap numbers suggested that I had plenty. No, not true! After scrubbing the code, it now works properly again. Web pages fly :-))

@westfw, thanks for your comments. However, a lot of that is already above my pay grade :frowning: . Do you have any links, where the tools you mentioned, are explained?

19

However, a lot of that is already above my pay grade :frowning: . Do you have any links, where the tools you mentioned, are explained?

Well, the tools have the usual man pages and manuals and stuff online (the utilities and I think all of the commands I mentioned are "generic" gcc tools, rather than anything specific to the ESP32.)

In general, it's obscure stuff and even if you have a high pay grade, it's someone else's job (compiler/tool support team) to understand it all. If you're lucky, after a decade or two of occasionally being told "oh, use this mysterious command", you develop a feel for what capabilities OUGHT to exist, so that you can searh the manuals and/or know reasonable questions to ask people with greater expertise...

Now, I'd think there would be a way to get the linker to dump the information that it knows about the target memory (from the linker scripts), in a way similar to the compiler command "gcc options -dM -E - < /dev/null" (which shows the symbols that are defined BEFORE any include files happen.) And I've asked on some other forums. So far, there are some answers that are close, but not quite what I want...

20

@westfw: I was typing a question while you were already posting the answer ...

Thanks for the consolidation :frowning:

So all I have to do is to wait for two decades, and then perhaps struggle with the then just-released ESP 9000, which will be able to talk to me and say “I'm sorry Ullix, I'm afraid I can't do that …”.

I found all the tools you mentioned, and I can use them, yet their answers are more than cryptic. I'll take a look at your new links.