ESP32 Repeatedly Crashes with FreeRTOS Task

Hello,
I've run into a problem recently when working on a new project involving an ESP32 and a max43421e(USB Host). There were relatively few problems regarding the hardware or the library(Host Shield Library 2.0 by Oleg Mazurov).

Problem:
I have an issue when I try to run everything in a FreeRTOS task instead of the setup and 'loop' functions. *Please bare with me, I'm still very new to FreeRTOS. I placed all the routines and declarations inside the task. This is when the ESP began to crash over and over.

Clue:
I noticed that when I define "USB Usb;" inside the task, it crashes. When I define "USB Usb;" outside of the task, globally, it works just fine. At first, I thought it must be some kind of stack overflow, but even when I increase the task's stack to 100's of KB's, it still keeps crashing.

Replicating The Problem:
To replicate the problem, you don't need the max43421e(USB Host). The sketch will behave similarly with or without the shield. I've only tested this on my ESP32. I have not yet tested it on my ESP8266

I just want to know why this is happening. Is it just a dumb mistake on my part or could it be revealing a larger underlying problem? Any help or insight will be greatly appreciated!

Below is the sketch that causes my ESP to crash repeatedly and the Crash reports/error logs that the ESP Generates. The sketch is a modified version of Oleg's "USBHIDBootKbd" example:

[edit]: The library can be found here: GitHub - felis/USB_Host_Shield_2.0: Revision 2.0 of USB Host Library for Arduino.

Sketch:

#include <hidboot.h>
#include <usbhub.h>
#include <SPI.h>

/*
 //////// Sketch Overview ////////
    class KbdRptParser
    {
      methods to help report keyboard events. This is handed over to 'HIDBoot' in 'usb_xTask()'
    }
    usb_xTask()
    {
      Create USB Objects
      Init Serial
      Init USB
      Run and report USB Tasks
    }
    setup()
    {
      Create USB Task
    }
    loop()
    {
    }

    KbdRptParser Implemented methods:
    ...
*/




// define 'KbdRptParser' class. Implementations are lower in the sketch
class KbdRptParser : public KeyboardReportParser
{
  void PrintKey(uint8_t mod, uint8_t key);

protected:
  void OnControlKeysChanged(uint8_t before, uint8_t after);

  void OnKeyDown(uint8_t mod, uint8_t key);
  void OnKeyUp(uint8_t mod, uint8_t key);
  void OnKeyPressed(uint8_t key);
};




// USB Usb; // << If 'Usb' is declared here, the sketch runs without crashing

void usb_xTask(void *__par__)
{
  // setup:
  USB Usb;  // <<< This is the issue, If declared globally, it works fine. If declared within this task, crashes after "USB::Task"
  HIDBoot<USB_HID_PROTOCOL_KEYBOARD> HidKeyboard(&Usb);
  KbdRptParser Prs;

  Serial.begin(115200);
  Serial.println("Start");

  // Check if host is communicating. Keep executing regardless, program crash happens either way
  if (Usb.Init() == -1)
    /* normally exit program or implement a way to wait for USB Shield to begin communicating
       For the sake of this example, report and keep executing: */
    Serial.println("OSC did not start.");
  else
    Serial.println("OSC Started");

  HidKeyboard.SetReportParser(0, &Prs); // Set parser for keyboard changes

  uint8_t usbTask = 255; // value for keeping track of Task State Changes



  Serial.println("Start Task");
  Usb.Task(); // just an extra 'Task' call to trigger a program crash
  Serial.println("End Task, Program didn't crash"); // << If 'Usb' was declared inside 'usb_xTask', the program will never reach this point


  // loop:
  for (;;)
  {
    Usb.Task(); // Run Usb Task. <<< This is were I believe the ESP crashes

    // simply get the usb task state and report changes:
    uint8_t ts = Usb.getUsbTaskState();
    if (usbTask != ts)
    {
      usbTask = ts;
      Serial.print("USB Task State: ");
      Serial.println(usbTask);
    }
    delay(10); // allow other tasks to run
  }
}



void setup() {
  // create task:
  xTaskCreate(
      usb_xTask,  // task name
      "USB Task", // task description
      2048,       // stack size. I don't think this is the issue. I raised the number to hundreds of KBs and still doesn't seem to resolve the issue
      NULL,       // task parameters
      1,          // task priority
      NULL);      // task handler
}

void loop() {delay(10);}



void KbdRptParser::PrintKey(uint8_t m, uint8_t key)
{
  MODIFIERKEYS mod;
  *((uint8_t *)&mod) = m;
  Serial.print((mod.bmLeftCtrl == 1) ? "C" : " ");
  Serial.print((mod.bmLeftShift == 1) ? "S" : " ");
  Serial.print((mod.bmLeftAlt == 1) ? "A" : " ");
  Serial.print((mod.bmLeftGUI == 1) ? "G" : " ");

  Serial.print(" >");
  PrintHex<uint8_t>(key, 0x80);
  Serial.print("< ");

  Serial.print((mod.bmRightCtrl == 1) ? "C" : " ");
  Serial.print((mod.bmRightShift == 1) ? "S" : " ");
  Serial.print((mod.bmRightAlt == 1) ? "A" : " ");
  Serial.println((mod.bmRightGUI == 1) ? "G" : " ");
};

void KbdRptParser::OnKeyDown(uint8_t mod, uint8_t key)
{
  Serial.print("DN ");
  PrintKey(mod, key);
  uint8_t c = OemToAscii(mod, key);

  if (c)
    OnKeyPressed(c);
}

void KbdRptParser::OnControlKeysChanged(uint8_t before, uint8_t after)
{

  MODIFIERKEYS beforeMod;
  *((uint8_t *)&beforeMod) = before;

  MODIFIERKEYS afterMod;
  *((uint8_t *)&afterMod) = after;

  if (beforeMod.bmLeftCtrl != afterMod.bmLeftCtrl)
  {
    Serial.println("LeftCtrl changed");
  }
  if (beforeMod.bmLeftShift != afterMod.bmLeftShift)
  {
    Serial.println("LeftShift changed");
  }
  if (beforeMod.bmLeftAlt != afterMod.bmLeftAlt)
  {
    Serial.println("LeftAlt changed");
  }
  if (beforeMod.bmLeftGUI != afterMod.bmLeftGUI)
  {
    Serial.println("LeftGUI changed");
  }

  if (beforeMod.bmRightCtrl != afterMod.bmRightCtrl)
  {
    Serial.println("RightCtrl changed");
  }
  if (beforeMod.bmRightShift != afterMod.bmRightShift)
  {
    Serial.println("RightShift changed");
  }
  if (beforeMod.bmRightAlt != afterMod.bmRightAlt)
  {
    Serial.println("RightAlt changed");
  }
  if (beforeMod.bmRightGUI != afterMod.bmRightGUI)
  {
    Serial.println("RightGUI changed");
  }
}

void KbdRptParser::OnKeyUp(uint8_t mod, uint8_t key)
{
  Serial.print("UP ");
  PrintKey(mod, key);
}

void KbdRptParser::OnKeyPressed(uint8_t key)
{
  Serial.print("ASCII: ");
  Serial.println((char)key);
};

ESP Crash Reports:

Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x400d2a7f  PS      : 0x00060e30  A0      : 0x800d1df8  A1      : 0x3ffb8930  
A2      : 0x3ffb8990  A3      : 0x3ffb8a30  A4      : 0x00000000  A5      : 0x3ffb8a08  
A6      : 0x3ffb8a48  A7      : 0x00000000  A8      : 0x800d2a84  A9      : 0x3ffb8900  
A10     : 0x3ffa9bbc  A11     : 0xcd8ae580  A12     : 0xcd8ae580  A13     : 0x00000000  
A14     : 0x000000ff  A15     : 0x3ffb8900  SAR     : 0x0000001b  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x400862c1  LEND    : 0x400862d1  LCOUNT  : 0xfffffffd  


Backtrace: 0x400d2a7c:0x3ffb8930 0x400d1df5:0x3ffb8950

Start with running the exception dump through the ESP Exception Decoder. Post the results.

Pretend we have no idea what that means.

Please post a GitHub link to those libraries.

So Sorry, my bad. I edited my post.
The Library can be found here. Just one library. freeRTOS seems to be built-in the ESP32 core: GitHub - felis/USB_Host_Shield_2.0: Revision 2.0 of USB Host Library for Arduino.

Here are the results from the decoder:

PC: 0x400d2a7f: USB::Task() (C:\Users\***\Documents\****\Arduino\libraries\USB_Host_Shield_Library_2.0\Usb.cpp line 497
EXCVADDR: 0x0000000c

Decoding stack results
0x400d2a7c: USB::Task() (C:\Users\***\Documents\****\Arduino\libraries\USB_Host_Shield_Library_2.0\Usb.cpp line 496
0x400d1df5: usb_xTask(void*) (C:\Users\***\Documents\****\Arduino\Projects\*****\ProjecterHost\tests\RTOS_UsbIssue/RTOS_UsbIssue.ino line 76

This is the line found exactly at line 496 of Usb.cpp:

        for(uint8_t i = 0; i < USB_NUMDEVICES; i++)
                if(devConfig[i])  //  <<<< line 496
                        rcode = devConfig[i]->Poll();

This is found inside void USB::Task(void)

Somehow that is causing an illegal read (thats what I'm assuming "LoadProhibited" means)

Line 76 in the main sketch is the call to Usb.Task():

  Serial.println("Start Task");
  Usb.Task(); // just an extra 'Task' call to trigger a program crash
  Serial.println("End Task, Program didn't crash"); // << If 'Usb' was declared inside 'usb_xTask', the program will never reach this point

One more thing I tried, I moved Everything inside setup(to see if the issue was due to the local scope of the functions) and it works just fine. The problem is specifically with the RTOS task

This is the code I used:

#include <hidboot.h>
#include <usbhub.h>
#include <SPI.h>

// define 'KbdRptParser' class. Implementations are lower in the sketch
class KbdRptParser : public KeyboardReportParser
{
    void PrintKey(uint8_t mod, uint8_t key);

  protected:
    void OnControlKeysChanged(uint8_t before, uint8_t after);

    void OnKeyDown(uint8_t mod, uint8_t key);
    void OnKeyUp(uint8_t mod, uint8_t key);
    void OnKeyPressed(uint8_t key);
};




void setup() {
  // setup:
  USB Usb;  // <<< Not an issue any more if declared and used inside setup
  HIDBoot<USB_HID_PROTOCOL_KEYBOARD> HidKeyboard(&Usb);
  KbdRptParser Prs;

  Serial.begin(115200);
  Serial.println("Start");

  // Check if host is communicating. Keep executing regardless, program crash happens either way
  if (Usb.Init() == -1)
    /* normally exit program or implement a way to wait for USB Shield to begin communicating
       For the sake of this example, report and keep executing: */
    Serial.println("OSC did not start.");
  else
    Serial.println("OSC Started");

  HidKeyboard.SetReportParser(0, &Prs); // Set parser for keyboard changes

  uint8_t usbTask = 255; // value for keeping track of Task State Changes



  Serial.println("Start Task");
  Usb.Task(); // just an extra 'Task' call to trigger a program crash
  Serial.println("End Task, Program didn't crash");


  // loop:
  for (;;)
  {
    Usb.Task(); // Run Usb Task. <<< This is were I believe the ESP crashes

    // simply get the usb task state and report changes:
    uint8_t ts = Usb.getUsbTaskState();
    if (usbTask != ts)
    {
      usbTask = ts;
      Serial.print("USB Task State: ");
      Serial.println(usbTask);
    }
    delay(10); // allow other tasks to run
  }
}

void loop() {
  delay(10);
}



void KbdRptParser::PrintKey(uint8_t m, uint8_t key)
{
  MODIFIERKEYS mod;
  *((uint8_t *)&mod) = m;
  Serial.print((mod.bmLeftCtrl == 1) ? "C" : " ");
  Serial.print((mod.bmLeftShift == 1) ? "S" : " ");
  Serial.print((mod.bmLeftAlt == 1) ? "A" : " ");
  Serial.print((mod.bmLeftGUI == 1) ? "G" : " ");

  Serial.print(" >");
  PrintHex<uint8_t>(key, 0x80);
  Serial.print("< ");

  Serial.print((mod.bmRightCtrl == 1) ? "C" : " ");
  Serial.print((mod.bmRightShift == 1) ? "S" : " ");
  Serial.print((mod.bmRightAlt == 1) ? "A" : " ");
  Serial.println((mod.bmRightGUI == 1) ? "G" : " ");
};

void KbdRptParser::OnKeyDown(uint8_t mod, uint8_t key)
{
  Serial.print("DN ");
  PrintKey(mod, key);
  uint8_t c = OemToAscii(mod, key);

  if (c)
    OnKeyPressed(c);
}

void KbdRptParser::OnControlKeysChanged(uint8_t before, uint8_t after)
{

  MODIFIERKEYS beforeMod;
  *((uint8_t *)&beforeMod) = before;

  MODIFIERKEYS afterMod;
  *((uint8_t *)&afterMod) = after;

  if (beforeMod.bmLeftCtrl != afterMod.bmLeftCtrl)
  {
    Serial.println("LeftCtrl changed");
  }
  if (beforeMod.bmLeftShift != afterMod.bmLeftShift)
  {
    Serial.println("LeftShift changed");
  }
  if (beforeMod.bmLeftAlt != afterMod.bmLeftAlt)
  {
    Serial.println("LeftAlt changed");
  }
  if (beforeMod.bmLeftGUI != afterMod.bmLeftGUI)
  {
    Serial.println("LeftGUI changed");
  }

  if (beforeMod.bmRightCtrl != afterMod.bmRightCtrl)
  {
    Serial.println("RightCtrl changed");
  }
  if (beforeMod.bmRightShift != afterMod.bmRightShift)
  {
    Serial.println("RightShift changed");
  }
  if (beforeMod.bmRightAlt != afterMod.bmRightAlt)
  {
    Serial.println("RightAlt changed");
  }
  if (beforeMod.bmRightGUI != afterMod.bmRightGUI)
  {
    Serial.println("RightGUI changed");
  }
}

void KbdRptParser::OnKeyUp(uint8_t mod, uint8_t key)
{
  Serial.print("UP ");
  PrintKey(mod, key);
}

void KbdRptParser::OnKeyPressed(uint8_t key)
{
  Serial.print("ASCII: ");
  Serial.println((char)key);
};

Hold on, I might have found the issue. I have to run more tests(compilation is taking forever between tests)

I will return with results

I found the Problem. Unfortunately, this problem did indeed reveal that there may be many more problems just like it throughout the library. This problem just happened to cause major crashes

[Note] I am using ver 1.6.2. of the library posted above

I will be posting my solution here just in case someone might find it useful one day. Big thank you to @gfvalvo for suggesting ESP Exception Decoder. I would have not been able to isolate the problem so fast otherwise!

The program crashes after attempting to invoke a method from a pointer inside devConfig[]. devConfig[] is an array of USBDeviceConfig*. It is defined in line 214 of UsbCore.h. I printed out the contents of devConfig[] in Usb.cpp right before the crash happens and found a seemingly random address pointer inside the array. The ESP crashes when it attempts to invoke a method from this random address.

After looking further into the code, I found that devConfig[] is never properly initialized and it happens to pick up random values left over in the stack memory. This in turn causes illegal read attempts when the pointer is invoked which causes program crashes.

To fix devConfig[], I simply initialized all values inside the USB constructor in Usb.cpp like so:

/* constructor */
USB::USB() : bmHubPre(0) {
        for(uint8_t i = 0; i < USB_NUMDEVICES; i++)  devConfig[i] = nullptr;  //  initialize array
        usb_task_state = USB_DETACHED_SUBSTATE_INITIALIZE; //set up state machine
        init();
}

This made the program run as expected and without crashing. However, aside from devConfig[], there may be more similar uninitialized variables that can cause unexpected behavior that I haven't run into just yet. One other bug I found so far was that the ESP only reports the keyboard strokes when the keyboard has been reinserted at least once

Is there any way to initialize the entire stack memory inside usb_xTask at the start-up of the task?

I recreated your results and was just starting to pull on that very thread.

That sounds like a unreliable and hacky solution. I'd start by bringing these problems to the attention of the library's author.

1 Like

BTW, which of these two print statements do you see with your hardware?

I get "OSC did not start" presumably because I don't have a max43421e connected. It's probably not valid to call Usb.Task() if the call to Usb.Init() fails.

No offense, but isn't that pretty obvious? That is the whole point of prividing a return value! If any init function fails, you have no ides what has, or has not, been properly initialized, so all bets are off, and you should NOT access any other functions or data.

Like I said, I believe the call to Usb.Init() failed in my setup because don't have the max43421e. I don't know if the call failed in OP's setup. So, I don't know if his failure mode was the same as mine. That's why I asked.

I tested both with and without the max43421e.
With the max43421e: "OSC Started".
Without the max43421e: "OSC did not start"

It does not cause any weird behavior when I ran it without the shield. It started as expected and began reporting the task state without crashing. Except that without the shield, it simply returns task state '18' with no further action.

It crashes exactly the same way, at the same line, with the same corrupted memory addresses with or without the shield.

As commented in my sketch:

/* normally exit program or implement a way to wait for USB Shield to begin communicating
       For the sake of this example, report and keep executing: */

Normally you would not call Usb.Task() without proper initialization. But just incase some don't have the shield, I wanted to make sure they would be able to recreate the problem

These are the logs for when the shield is connected and the ESP crashes:

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1344
load:0x40078000,len:13924
ho 0 tail 12 room 4
load:0x40080400,len:3600
entry 0x400805f0
Start
OSC Started
Start Task
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x400d2a7f  PS      : 0x00060e30  A0      : 0x800d1df8  A1      : 0x3ffb8930  
A2      : 0x3ffb8990  A3      : 0x3ffb8a30  A4      : 0x00000001  A5      : 0x3ffb8a08  
A6      : 0x3ffb8a48  A7      : 0x00000000  A8      : 0x800d2a84  A9      : 0x3ffb8900  
A10     : 0x3ffa9bbc  A11     : 0xf9e2b2e6  A12     : 0xf9e2b2e6  A13     : 0x00000000  
A14     : 0x000000ff  A15     : 0x3ffb8900  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x4008409d  LEND    : 0x400840a5  LCOUNT  : 0x00000027  


Backtrace: 0x400d2a7c:0x3ffb8930 0x400d1df5:0x3ffb8950

These are the logs for when the Shield is not connected and crashes:

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1344
load:0x40078000,len:13924
ho 0 tail 12 room 4
load:0x40080400,len:3600
entry 0x400805f0
Start
OSC did not start.
Start Task
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x400d2a7f  PS      : 0x00060e30  A0      : 0x800d1df8  A1      : 0x3ffb8930  
A2      : 0x3ffb8990  A3      : 0x3ffb8a30  A4      : 0x00000000  A5      : 0x3ffb8a08  
A6      : 0x3ffb8a48  A7      : 0x00000000  A8      : 0x800d2a84  A9      : 0x3ffb8900  
A10     : 0x3ffa9bbc  A11     : 0x622bded9  A12     : 0x622bded9  A13     : 0x00000000  
A14     : 0x000000ff  A15     : 0x3ffb8900  SAR     : 0x0000001b  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x400862c1  LEND    : 0x400862d1  LCOUNT  : 0xfffffffd  


Backtrace: 0x400d2a7c:0x3ffb8930 0x400d1df5:0x3ffb8950

The decoder returns same exact results with and without the shield

[note] When there is no shield connected, the only difference I noticed, is the Usb.Init() call takes significantly longer as it is attempting to communicate with a non-existent shield

Can I have your opinion on this kind of 'hacky' solution? It seems to fix the problem. I just don't know if its safe:

#include <new>
.
.
.
void usb_xTask(void *__par__)
{
  // setup:
  USB *Usb = (USB*)malloc(sizeof(USB));  // << allocate memory
  if(!Usb) while(1);  // << ERROR!!!
  memset(Usb, 0, sizeof(USB));  // << initialize allocated memory
  new(Usb) USB();  // << create instance of class


  HIDBoot<USB_HID_PROTOCOL_KEYBOARD> *HidKeyboard = (HIDBoot<USB_HID_PROTOCOL_KEYBOARD>*)malloc(sizeof(HIDBoot<USB_HID_PROTOCOL_KEYBOARD>));  // << allocate memory
  if(!HidKeyboard) while(1);  // << ERROR!!!
  memset(HidKeyboard, 0, sizeof(HIDBoot<USB_HID_PROTOCOL_KEYBOARD>));  // << initialize allocated memory
  new(HidKeyboard) HIDBoot<USB_HID_PROTOCOL_KEYBOARD>(Usb);  // << create instance of class

  KbdRptParser Prs; // << no need to initialize memory since no variables are present in this class

  .
  .
  .

  // destruct objects:
  Usb->~USB();
  HidKeyboard->~HIDBoot<USB_HID_PROTOCOL_KEYBOARD>();

  // free memory:
  free(Usb);
  free(HidKeyboard);
  Usb = nullptr;
  HidKeyboard = nullptr;
  vTaskDelete(NULL);
  //  return;
}

Actually, I misread your proposal in your previous post. Now I understand what you intended. This seems like it would work, but I still think it would be preferable for the class to initialize itself properly.

1 Like

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.