Vsnprintf() random core panic

Projects Programming
1

Hello (sorryI don't know which category it should be posted in),

Did anyone experienced random crashes and core panic' on ESP32 platform using vsnprintf() ? The code below crashes almost always if buffer length is 128 and never when it is 80. Got a number of different exceptions but they are related to stack overrun. Yes I tried to increase stack sizes.

void
xprintf(const char* format, ...) {
	
  va_list args;
  va_start(args, format);
  
  char buf[128];
  vsnprintf(buf, 128, format, args);
  
  va_end(args);

  Serial.print(buf);
}

This code is called from two different tasks, once per minute.

If i comment Serial.print(buf); out it still crashes. The format strings I supply to it is nothing special and they are all below. Types match formats.

xprintf("CPU : %score, APP is CPU%d, PRO is CPU%d\n", portNUM_PROCESSORS > 1 ? "multi" : "single",app_core, pro_core);
xprintf("Temp : Pin%d=%u, Voltage=%f, Speed=%f, next poll in %d seconds\n",TEMPERATURE_PIN,raw,voltage,sea_temp,TEMPERATURE_INTERVAL);
xprintf("Wind: Pin%d=%u, Voltage=%f, Speed=%f, next poll in %d seconds\n",WINDSPEED_PIN,raw,voltage,speed,WINDSPEED_INTERVAL);

When I change the buffer size to 80 then it all works as expected.

2

I think the question fits better in the Programming Questions section and hence it has moved there.

3

When it crashes is see this:

load:0x40080404,len:3356
entry 0x4008059c
CPU : multicore, APP is CPU1, PRO is CPU0
CPU#0 : Task "Temperature" started
CPU#0 : Task "Windspeed" started
Guru Meditation Error: Core  0 panic'ed (Unhandled debug exception). 
Debug exception reason: Stack canary watchpoint triggered (IDLE0) 
Core  0 register dump:
PC      : 0x4008944b  PS      : 0x00060036  A0      : 0x4008a88f  A1      : 0x3ffb89a0  
A2      : 0x3ffc1d6c  A3      : 0x3ffc1f60  A4      : 0x00000001  A5      : 0x3ffc1f60  
A6      : 0x00000000  A7      : 0x00000000  A8      : 0x3ffc1d74  A9      : 0x3ffb8980  
A10     : 0x3ffbc6e0  A11     : 0x3ffbc6e0  A12     : 0x00000014  A13     : 0x3ffbc6dc  
A14     : 0x80000001  A15     : 0xb33fffff  SAR     : 0x00000020  EXCCAUSE: 0x00000001  
EXCVADDR: 0x00000000  LBEG    : 0x400861f9  LEND    : 0x40086209  LCOUNT  : 0xffffffff  


Backtrace: 0x40089448:0x3ffb89a0 0x4008a88c:0x3ffb89d0 0x4008a83c:0xa5a5a5a5 |<-CORRUPTED

However it happens not always and not often. I played with intervals (there are 2 tasks on Core0) and it looks to me as it related to multitasking. Are vsnprintf reentrant?

It crashes randomly. SOmetimes it works for long time, sometimes it crashes immediately.

I can post the complete source code but it is not just 1 .ino file. It is a C library and .ino file.

4

vsnprintf_crash.zip (5,8 КБ)

complete source code. all the .c code is moved to a library because I found no way how I can add multiple .h and .c files into my project directly. Timing intervals were set below 1 sec just for testing. It crashes when it set to integers as well. I was just trying to make it happen more often

5

Did you write the library ? I don't like it.
When everything is put into a single file, and stripped from everything that makes it complex, then it should work.
Always follow the KISS rule: https://en.wikipedia.org/wiki/KISS_principle

I can not find the xprintf() calls in the zip file that you show in your post. I can not find the 'raw' variable. It is important that you show the code with the problem.

Using a variable argument list with defines can easily go wrong without warning, try to avoid that. Convert them first.

#define NUMBER_FLOWERS 3
int nFlowers = NUMBER_FLOWERS;
printf("%d", nFlowers);

Some say that va_list and va_copy should be used.
Is vsnprintf() safe to use in a multitasking system ?

Do you have the classic ESP32 ?
Then the WiFi things runs on Core0, and Arduino runs on the Core1. Keep all your code on Core1 or just do xTaskCreate to be compatible with single core processors.
Please don't use HardwareSerial or something. The "Serial", "Serial1" and "Serial2" are already available for you.
I don't know if 2048 is enough for the stack.

6
  1. Yes it is my library
  2. 'raw' is defined in the function body:
static void
temperature_task(void *arg) {

	unsigned int raw;

and

static void
windspeed_task(void *arg) {

	float voltage;
	unsigned int raw;

I program in C.
I need a wrapper to call Serial.printf(...) from the C code.

And yes, this is the question: are vsnprintf/snprintf/etc multitask safe on ESP32 platform?

7

Classic ESP32, dualcore.
ESP32-WROOM-32D board, 30 pin devboard, CPU: ESP32-D0WD Ver3

8

it is defined in sketch.ino ; it is called from temperature.c windspeed.c task.c and sketch.ino as well.

9

Can you please just unpack the archive to your sketch folder, restart your Arduino IDE and load a sketch named "sketch"? It is acomplete code. Nothing else needed to check the issue. Just plain generic ESP32 dual core board.

Sometimes it runs ok, but then after couple of resets (pressing a "Reset" button) it gives the exception

When I modify my xprintf() to use malloc'ed memory then it works as it should. But with buffer on stack it crashes. For me it looks as a multitask/sync bug.

10

Many people here will not open a sketchy .zip file of unknown provenance. If there are too many files to post inline on the forum, make a GitHub repository of your project.

11

Oh.
Attached files should go in [SketchesDirectory]/libraries/Weather/
config.h (734 байта)
task.c (1 КБ)
task.h (736 байтов)
temperature.c (1,7 КБ)
temperature.h (550 байтов)

12

weather.h (439 байтов)
windspeed.c (1,4 КБ)
windspeed.h (534 байта)

13

The actual sketch goes into [SketchesDirectory]/sketch/
sketch.ino (479 байтов)

14

My guess would be the 128 char stack-based local buffer is over-running the task stack.

15

I would try making the 128 char buffer global, and see if the problem goes away. If it does, that is a good indication the stack is overflowing.

16

Well.. it goes away if the buffer is not on stack yes. But it also can be caused by two task accessing the same buffer somewhere inside the vsnprintf. To be honest I can't make it global because of that. Or I have to use semaphores which I don't want to. I tried to memset() this buffer with 0x00 instead of vsnprintf and it works ok. Right now I have a workaround: malloc()/free() that buffer.

17

I think that this is the same as your code.

// Forum: https://forum.arduino.cc/t/vsnprintf-random-core-panic/1286637
// This Wokwi project: https://wokwi.com/projects/404875843203421185

// Enable or disable the WDEBUG
#define WDEBUG

// The maximum wind for 3.3V.
#define SENSOR_MAX 30.0

// Temperature / Windspeed value indicating a sensor failure.
// A value above 5000 is invalid.
#define INVALID (6000.0)        

// ESP32 board pin where LM35DZ sensor is connected
const int temperaturePin = 33;

// SN-30000-FSJT-V05 windspeed sensor is connected
const int windspeedPin = 34;

// Global variables that are updated in the tasks.
volatile double windspeed   = INVALID;
volatile double temperature = INVALID;

void setup() 
{
  Serial.begin(115200);
  Serial.println("=======================================================");

  xTaskCreate(windspeed_task  , "Wind"       , 8000, NULL, 1, NULL);
  xTaskCreate(temperature_task, "Temperature", 8000, NULL, 1, NULL);
}

// The loop() is a FreeRTOS task,
// running at priority 1.
void loop() 
{
  noInterrupts();
  double wCopy = windspeed;
  double tCopy = temperature;
  interrupts();

  Serial.printf("LOOP windspeed = %.2lf, temperature = %.2lf\r\n", wCopy, tCopy);
  delay(2000);
}

void windspeed_task(void *arg) 
{
  while(1) 
  {
    int raw = analogRead(windspeedPin);
    double v = (double) raw / 4096.0 * 3.3;
    double s = v / 3.3 * SENSOR_MAX;

    // update the global variable
    windspeed = s;

#ifdef WDEBUG
    Serial.printf("WIND windspeed = %.2lf                      (voltage = %.2lf)\r\n",s,v);
#endif

    delay(5000);
  }
}

void temperature_task(void *arg) 
{
  while(1) 
  {
    int raw = analogRead(temperaturePin);
    double v = (double)raw / 4096.0 * 3.3; 
    double t = v * 100.0; // 0.32v = 32 degrees Celsius

    // update the global variable
    temperature = t;

#ifdef WDEBUG
    Serial.printf("TEMP                   temperature = %.2lf (voltage = %.2lf)\r\n",t,v);

    // Anything above 35 degrees is suspicous but OK.
    // Above 99 is steam.
    if(t > 100.0)
      Serial.printf("TEMP Sensor failure?\r\n");
#endif

    delay(3000);
  }
}

Try it in Wokwi:

After starting the simulation, move the sliders with the mouse.

Some changes are common in the Arduino world, some changes are my own preferences. I can try to mention a few:

  • So many files for so little code does make it difficult. I keep clicking on the tabs.
  • Please use spaces, not tabs. Tabs are never the same size.
  • Optimizing for 'float' instead of 'double' is not needed.
  • Arduino prefers to put pin numbers in variables. Defines can be overwritten, therefor variables are safer.
  • There is absolutely no need to split tasks over different cores.
  • The usage of vsnprintf() has been changed over the years. Avoid it.
  • There is no need to create variables in the startup section of the task. They can be local inside the while-statement.

[UPDATE] Now that I think about it, turning of the interrupts in the loop() is not enough to get valid data in a pre-emptive multitasking system. The sketch is therefor not good enough.

18

Well, it's up to you to make sure that can NEVER happen! That's why semaphores are needed.

19

Well what I found so far is that vsnprintf() is not reentrant on an ESP32 dualcore. At least on a startup.

It crashes even with strings like xprintf("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); with no formatting at all.

And it is somehow linked to task startup: once task createion is delayed then bug fades away. Thats sad.

[offtopic]
Regarding tabs and many files: this is how I make my code, this is what I did
when I was working in software companies. Kinda standart which I get used to. And we were punished for using spaces and mixing different code into single files :). Just look into IDF code and blame them then. Tab sizes can be configured in your editor, spaces are not, thats why. I prefer tabs with 4 spaces per tab. There are not so many files tho. It is just a beginning, not a finished code. For me it is easier to navigate thru bunch of files instead of one single file.

Regarding two cores: yes it could be done on a single core, I just wanted to play with multicore system and hit this bug.
[/offtopic]

I thought that arduino uses a newlib, and newlib must have a reentrant versions of vsnprintf. Seems like it doesn't

20

It must not happen on reentrant versions.