AsyncWebSocket::textAll memory leak using ESP32?

Projects Programming
1

Hey everyone,

I have been working on a project of mine, moving my current Arduino based motor test (for my remote control places) into ESP32 with some enhanced functionality to be able to do all tests through my tablet or PC and retain all the results for later analysis.

The concept here is to run a series of tests on the motor with different throttle power and get readings for battery voltage amps, thrust of the motor etc. All operations for the app are via REST calls to the ESP32 and all results and sensor readings are using a Web Socket opened between the ESP32 and the App (primarily one way).

For this I have been using the ESPAsyncWebServer library to handle REST APIs and the WebSocket., but I am facing an issue for over 5 days now where the AsyncWebSocket::textAll method crashes forcing the ESP32 to abort the thread and restart. The exception Decoder gives me the below trace:

Decoding stack results
0x40083771: panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c line 408
0x4008be51: esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c line 137
0x4009149d: abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/abort.c line 46
0x40145d1f: __cxxabiv1::__terminate(void (*)()) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc line 47
0x40145d66: std::terminate() at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc line 57
0x40145f55: __cxxabiv1::__cxa_allocate_exception(std::size_t) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_alloc.cc line 300
0x4014600c: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_op.cc line 54
0x40146835: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_opv.cc line 32
0x400d9142: AsyncWebSocketMessageBuffer::AsyncWebSocketMessageBuffer(unsigned char*, unsigned int) at .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.cpp line 138
0x400d945d: AsyncWebSocket::makeBuffer(unsigned char*, unsigned int) at .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.cpp line 1210
0x400d9d9c: AsyncWebSocket::textAll(char const*, unsigned int) at .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.cpp line 962
0x400d9db9: AsyncWebSocket::textAll(char const*) at .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.cpp line 1095
0x400d5d72: streamSensorValues() at src/main.cpp line 281
0x400d5d9c: webServerHandler(void*) at src/main.cpp line 137

For your info, I am using 2 tasks between the 2 cores, one to continuously read sensor data and the other one to serve any REST and WebSocket operations.

    xTaskCreatePinnedToCore(processSensors, "Read sensor values", 10000, NULL, 1, &TaskProcessSensors, 0);
    xTaskCreatePinnedToCore(webServerHandler, "REST API and Web Streaming", 10000, NULL, 1, &TaskWebServerHandler, 1);

void processSensors(void * pvParameters)
{
    for(;;)
    {
        if (mt.IsActive() && (int) (millis() - timer1) > refreshRate / 2)
        {
            // read sensor values twice within the refresh period
            mt.Handle();
            sensors = mt.GetReadings();
            timer1 = millis();
        }
        vTaskDelay(10 / portTICK_PERIOD_MS); 
    }
}

void webServerHandler(void * pvParameters)
{
    for(;;)
    {
        if (mt.IsActive() && (int) (millis() - timer2) > refreshRate)
        {
            streamSensorValues();
            timer2 = millis();
        }
        vTaskDelay(10 / portTICK_PERIOD_MS); 
    }
}

streamSensorValues will only be called if there is an active test and an opened WebSocket. this looks like this:

void streamSensorValues()
{
    jsonDocument.clear();
    jsonDocument["runningTime"] = mt.GetRunningTime();
    jsonDocument["phase"] = TestModes[sensors.SensorValues.Phase];
    jsonDocument["throttle"] = sensors.SensorValues.Throttle;
    jsonDocument["voltage"] = sensors.SensorValues.Voltage;
    jsonDocument["current"] = sensors.SensorValues.Current;
    jsonDocument["thrust"] = sensors.SensorValues.Thrust;
    jsonDocument["power"] = sensors.SensorValues.Power;
    jsonDocument["consumption"] = sensors.SensorValues.Consumption;
    if (serializeJson(jsonDocument, buffer) > 0) 
        ws.textAll(buffer);  // Stream sensor values
}

What I have noticed during all the tests I have done so far, ESP32 will restart roughly after 2min and 15s within a test, when roughly 1053 readings have been gone through the WebSocket. the time varies if I change the refresh rate of the entries sent out in the WebSocket but the total number of entries is roughly the same (between 1040 to 1130 sometimes).

I can also confirm that the cause is not the jsonDocument operations, because I get the same results if I sent a static text with roughly the same size as the readings being streamed, and it's all down to the number of entries I send over the WebSocket.

I have been reading a lot of posts in various forums so far and I have seen the occasional post indicating that the AsyncWebSocket::textAll has some memory leak issues, but the posts have been about 3-4 years old and I was wondering if anyone is experiencing something similar or even better if anyone has any idea how I can fix the issue :slightly_smiling_face:

Looking forward on any thoughts out there

Mike

2

you haven't posted your complete sketch.
How should anybody have a close look where the bug might be?

Do you make intensive use of Strings?
They eat up all RAM-memory over time

So far I have never used two cores and never encountered any problems doing so.
One core for the internal WiFi-stuff and whatever else is internally going on
One core for my code

Are you sure that using two cores is a must in your project?

3

Given that each of @michael_michailidis's task contain a 10ms on every pass, I'd say using both cores is definitely not required.

4

Thanks @StefanL38 for your feedback. All valid and good question :slightly_smiling_face:

First of all the project is in platformio, and quite large. Happy to post it though :slightly_smiling_face:

The thought behind the utilisation of both cores is because at an early stage I noticed some delays in the readings and processing of the sensor readings. I think I have now improved efficiency within code so you maybe right that 2 cores may be an overkill.

This however put me to thoughts so I tried the whole thing in a single core. Some improvements have been noticed no doubt, but still the same crash. I have now managed to up the time from 2m15s to 5min7s after an initial run.

Your comment about Strings is also correct, there is some hefty use of them, maybe during the generation of the simulated values. I also wonder if there is some fragmentation due to this as well. In this case, what would be a good (and easy if possible) way to monitor the Heap during a run?

5

I did a quogling and found

Serial.println(ESP.getFreeHeap());

If the Strings are causing the crash you could use SafeString.h instead.
SafeStrings are based on array of char (based on c_strings) but with almost the same comfort as Strings.

The name is program: you define them with a maximum length and because the lengths is defined at compile-time the RAM SafeStrings occupy stays always the same

1 Like
6

Here is a demo-code for using SafeStrings

// SafeStrings are based on array of chars
// SafeStrings offer almost the same comfort as Strings
// but avoid some dis-advantages of variable-type String
// the name SafeString is PROGRAM They are safe to use

// with the alternatives "String" must not always but CAN
// eat up memory over time which will make the code crash

// with zero-terminated array of chars (c_string)
// you have to take care of boundary checking yourself
// otherwise your code will start to behave weird and this kind of bug is very hard to find
// you can read more about this here https://www.forward.com.au/pfod/ArduinoProgramming/ArduinoStrings/index.html#safestring
// and here https://hackingmajenkoblog.wordpress.com/2016/02/04/the-evils-of-arduino-strings/

// very basic demo-code to show how to declare a SafeString-variable
// and how to assign text to them

// as a personal convention I use the suffix "_SS" to indicate
// THIS IS A SAFESTRING-VARIABLE
// but you can name it whatever you like

#include "SafeString.h"
createSafeString(myTitle_SS, 64);  // reserve 64 bytes for the SafeString-variable
createSafeString(myString_SS, 64); // reserve 64 bytes for the SafeString-variable
createSafeString(myExtraloongString_SS, 128); // reserve 128 bytes for the SafeString-variable

int myInteger = -1234;
float myFloat = -987.009;
float myFloat2 = -12345.6789;
float myFloat3 = -1234.567;


void setup() {
  Serial.begin(115200);
  Serial.println( F("Setup-Start") );
  Serial.println();

  myString_SS = F("fixed text directly assigned");
  Serial.print( F(" #") ); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println(F("#") ); // trailing double-cross "#" to show where the string REALLY ends


  myTitle_SS = F("content of an integer:");
  myString_SS = myInteger;

  Serial.println(myTitle_SS);
  Serial.print( F(" #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends


  myTitle_SS = F("content of a float:");
  myString_SS = myFloat;

  Serial.println(myTitle_SS);
  Serial.print( F(" #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

  myTitle_SS = F("you can append more text with the +=-operator ");
  myString_SS = F("text ");
  myString_SS += myInteger;
  myString_SS += F(" ,");
  myString_SS += myFloat;

  Serial.println(myTitle_SS);
  Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

  myString_SS += F(" ,");
  myString_SS += myFloat2;

  myString_SS += F(" ,");
  myString_SS += myFloat3;

  Serial.println(myTitle_SS);
  Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

  Serial.println( F(" now let's try to assign wayyyyyy too many characters 012345678900123456789001234567890012345678900123456789001234567890") );
  myString_SS  = "";
  myString_SS += F(" now let's try to assign wayyyyyy too many characters 012345678900123456789001234567890012345678900123456789001234567890");
  Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

  Serial.println( F(" now let's try to add more and more characters at the end of the SafeString") );
  myString_SS  = "";
  for (byte i = 10; i < 64; i++) {
    myString_SS += i;
    Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
    Serial.print(myString_SS);
    Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends
  }

  Serial.println( F(" assigning a number of chars the SafeString can really hold") );
  myExtraloongString_SS = F("less than 64 characters");
  myString_SS  = myExtraloongString_SS;
  Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

  
  Serial.println( F(" assigning too many chars than the SafeString can really hold") );  
  Serial.println( F("more than 64 characters 1234567890123456789012345678901234567890123456789012345678901234567890") ); 
  //                                  10        20        30        40        50        60        70
  //                         1234567890123456789012345678901234567890123456789012345678901234567890
  myExtraloongString_SS = F("more than 64 characters 1234567890123456789012345678901234567890123456789012345678901234567890");
  myString_SS  = myExtraloongString_SS;
  Serial.print( F("result: #" )); // leading double-cross "#" to show where the string starts
  Serial.print(myString_SS);
  Serial.println( F("#") ); // trailing double-cross "#" to show where the string REALLY ends

}

void loop() {
}
7

That may be the case, but it doesn't have to.
Strings can cause memory fragmentation as all dynamic memory usage can.
If there is 1 or more global Strings that get added onto while local Strings are created and destroyed, the memory can get fragmented. On an ESP32 though the chances of this actually causing a crash are not that big.

And rarely that soon.

As i said. As long as a String gets destroyed before a global String is being added onto, no fragmentation will occur. If you do use 'global' Strings, you can reserve the maximum space they may occupy using the .reserve() function.

Keep in mind there may also be other global objects and even local objects that have a similar influence. Still memory on an ESP32 is huge.

Yep ! You could even perform some kind of pause thing with the result of the available memory,

Or start blinking a led when half of it has been used or something.

8

Thank you all for your help on this issue.

I have now managed to identify the culprit :slight_smile:

The issue was that deep down the mt.Handle() method and when the Simulation values were set I was instantiating an object that controls calibration values , but was never destroying it. The calibration object should have been instantiated during the creation of the mt object once and used thereafter.

It looks that by doing the latter, the Heap usage has now stabilised to the expected value and I have managed to run a test for over 3.5 hours successfully :smiley:

Again, thanks to everyone who commented in this topic, it has pointed me to the right direction to resolve this

Mike

9

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.