ESP32: a better way than vTaskDelay to get around watchdog crash?

Hi, it's me again with more stupid questions.

I promise this one is definitely about dual core issues and not my crappy array management.

--

So I have a big pile of spaghetti here (link to sketch dump).

Most of it is functions related to controlling a nextion screen via serial and stepper motors. Down at the very bottom you'll see two core task assignments - one for the stepper loop, one for the screen loop.

Initially, I'd get repeating Task Watchdog restarts like this:

17:11:14.429 -> E (33316) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
17:11:14.429 -> E (33316) task_wdt:  - IDLE0 (CPU 0)
17:11:14.429 -> E (33316) task_wdt: Tasks currently running:
17:11:14.429 -> E (33316) task_wdt: CPU 0: Motors
17:11:14.429 -> E (33316) task_wdt: CPU 1: loopTask
17:11:14.463 -> E (33316) task_wdt: Aborting.
17:11:14.463 -> abort() was called at PC 0x400dd5f3 on core 0
17:11:14.463 -> 
17:11:14.463 -> Backtrace: 0x40088dd8:0x3ffbe160 0x40089005:0x3ffbe180 0x400dd5f3:0x3ffbe1a0 0x40081675:0x3ffbe1c0 0x40083087:0x3ffb4630 0x400810f7:0x3ffb4650 0x400d45ac:0x3ffb4670 0x400d47dd:0x3ffb4690 0x400d30f4:0x3ffb46b0 0x400d31b9:0x3ffb46d0 0x40087925:0x3ffb46f0
17:11:14.463 -> 
17:11:14.463 -> Rebooting...

So I did some reading and found others addressing the issue with a vTaskDelay instance with 10ms in their task loops. I stuck a couple in, and found that while it worked for most of the program's operation, when one of the cores entered a conditional loop [while (so and so)], I'd crash again. I stuck some more vTaskDelays in those loops and it did the trick...

but the accelstepper loops absolutely cannot have delays in them. The motors slow down to a useless crawl, doesn't matter how long the taskdelay is, it absolutely will not do.

So now I'm just researching and trying to find a way to address this - it seems to be an Arduino IDE-specific problem, and I did see one person suggesting to disable Task Watchdog. I don't know if that's a good idea, or even how to go about it, so I figured I'd better ask smarter people..

Any ideas here?

Thank you

Just a quick update: I did some more reading and discovered that there's functions for disabling the watchdog, so on setup I decided to disable it on both cores like so:

  disableCore0WDT();
  disableCore1WDT();

This worked, but when I got to my stepper motor loop (while statement containing stepper1.run() / stepper2.run()), they simply don't work.

I then tried only disabling core 0 (the stepper core) and it functioned the same, so I'm guessing that's the source of all this.

I can't get the stepper loop running with watchdog enabled, and it doesn't work with it disabled either apparently... :expressionless:


For clarity's sake, here's the problem loop function (running in a task loop for core 0):

void blockWinder() {

    stepper1.enableOutputs();
    stepper2.enableOutputs();
    s2_zero();
    s2_bobComp();
    
  for (int i = 0; i < 20; i ++) {
    stepper1.setCurrentPosition(0);

    stepper1.setMaxSpeed(MaxSpeed1);
    stepper1.setAcceleration(Accel1);

    stepper2.setMaxSpeed(MAX_SPD);
    stepper2.setAcceleration(MAX_XCL);
    stepper2.moveTo(Move2_A[i]);
    stepper2.runToPosition();
    
    stepper2.setMaxSpeed(MaxSpeed2[i]);
    stepper2.moveTo(Move2_B[i]);

    if (dirCW != true) {Move1[i] = -Move1[i];}
    
    stepper1.moveTo(Move1[i]);

    
        if (dirCW != true) {

        while (stepper1.distanceToGo() < 0) {
//Watchdog reboots here when enabled, and refuses to work with my steppers when disabled :'(
          stepper1.run();
        if (stepper2.currentPosition() == Move2_A[i]) {
          stepper2.moveTo(Move2_B[i]);}
        if (stepper2.currentPosition() == Move2_B[i]) {
          stepper2.moveTo(Move2_A[i]);}
        stepper2.run();
        }

        } else {

        while (stepper1.distanceToGo() > 0) {
//same in this loop, it's just the same thing but with a stepper driving the opposite direction
          stepper1.run();
        if (stepper2.currentPosition() == Move2_A[i]) {
          stepper2.moveTo(Move2_B[i]);}
        if (stepper2.currentPosition() == Move2_B[i]) {
          stepper2.moveTo(Move2_A[i]);}
          stepper2.run();
        };

      }

    
  };
  
  stepper2.moveTo(0);
  stepper2.runToPosition();
  windStatus = false;
  stepper1.disableOutputs();
  stepper2.disableOutputs();
}

Logically, comment out the instructions in task Motors, logically add back a few instructions at a time till you find the instruction group that causes the wdt, and then take troubleshooting the issue from there.

The function void blockWinder() is not a properly formatted freeRTOS task.

Idahowalker:
The function void blockWinder() is not a properly formatted freeRTOS task.

It's referenced from this task loop:

void MotorsTc( void * pvParameters ){
  Serial.print("Motors running on core ");
  Serial.println(xPortGetCoreID());

  for(;;){
while (windStatus == true) {
   blockWinder();
  };
  }
}

Is that a proper way to do it?

void MotorsTc( void * pvParameters ){
  Serial.print("Motors running on core ");
  Serial.println(xPortGetCoreID());

  for(;;){
while (windStatus == true) {
   blockWinder();
  };
  }
}

That task is running as fast as freeRTOS can run a task. The task does not contain any traffic control such as an event trigger or a vTaskDelay() or vTaskDelayUntil() macro. Meaning that task does not yield.

Here is an example of a freeRTOS task using a DelayUntill:

void fLIDAR_Is_OK ( void *pvParameters )
{
  String sSerial = "<$>";

  TickType_t xLastWakeTime;
  const TickType_t xFrequency = pdMS_TO_TICKS( 1000 );
  // Initialise the xLastWakeTime variable with the current time.
  xLastWakeTime = xTaskGetTickCount();
  for ( ;; )
  {
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
    bool LIDAR_OK = true;
    if ( xSemaphoreTake( sema_SendSerialToBrain, xZeroTicksToWait ) == pdTRUE ) // grab semaphore, no wait
    {
      if (xHANDLE_LIDAR_ServoAspectChange == NULL )
      {
        Serial.println ( " xHANDLE_LIDAR_ServoAspectChange " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_DoLIDAR == NULL )
      {
        Serial.println ( " xHANDLE_DoLIDAR " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_SendLIDAR_InfoSerialToBrain == NULL )
      {
        Serial.println ( " xHANDLE_SendLIDAR_InfoSerialToBrain " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_SendSerialToBrain == NULL )
      {
        Serial.println ( " xHANDLE_SendSerialToBrain " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_TweakServoX == NULL )
      {
        Serial.println ( " xHANDLE_TweakServoX " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_TweakServoY == NULL )
      {
        Serial.println ( " xHANDLE_TweakServoY " );
        LIDAR_OK = false;
      }
      if ( xHANDLE_GetIMU == NULL )
      {
        Serial.println ( " xHANDLE_GetIMU " );
        LIDAR_OK = false;
      }
      if ( LIDAR_OK )
      {
        SerialBrain.println ( sSerial );
      }
    }
    xSemaphoreGive ( sema_SendSerialToBrain );
    xLastWakeTime = xTaskGetTickCount();
  }
  vTaskDelete( NULL );
}

See how the task is not free running? Also, not the vTaskDelete( NULL ); which lies outside the run forever loop. If the vTaskDelete( NULL ); is run, something went wrong, a proper cleanup is initialized.

Here is a task using vTaskDelay:

void fBlinkBuiltIn( void* pvParameters )
{
  // toggle built in LED off/on
  for (;;)
  {
    vTaskDelay( pdMS_TO_TICKS( 10 ) );
    REG_WRITE( GPIO_OUT_W1TC_REG, BIT2 ); // GPIO2 LOW (clear)
    vTaskDelay( pdMS_TO_TICKS( OneK ) );
    REG_WRITE( GPIO_OUT_W1TS_REG, BIT2 ); //GPIO2 HIGH (set)
  }
  vTaskDelete( NULL );
}

Here is a task using an event

void fDoLIDAR( void * pvParameters )
{
  int iLIDAR_Distance = 0;
  while (1)
  {
    xEventGroupWaitBits (eg, evtDoLIDAR, pdTRUE, pdTRUE, portMAX_DELAY) ;
    tfmini.externalTrigger();
    iLIDAR_Distance = tfmini.getDistance();
    if ( iLIDAR_Distance > LIDAR_Max_Distance )
    {
      Serial.print ( " TFMini distance issue " );
      Serial.println (  iLIDAR_Distance );
    }
    xSemaphoreTake( sema_LIDAR_INFO, xSemaphoreTicksToWait );
    x_LIDAR_INFO.Range[x_LIDAR_INFO.CurrentCell] = iLIDAR_Distance;
    xSemaphoreGive( sema_LIDAR_INFO );
    xEventGroupSetBits( eg, evtLIDAR_ServoAspectChange );
  }
  vTaskDelete( NULL );
}

I did have vTaskDelays in the loops feeding TWD before, and it worked just fine until it gets to that loop with the stepper1.run() / stepper2.run() functions in it. I mean, the delay will work in there, but that loop needs to be moving as fast as can be, otherwise the motors are just stepping pitifully slow.

I can't figure out how to get around that issue - apparently it needs watchdog to to function, but functions like flaming diarrhea with it.


e: ...can I feed the core0 watchdog from core1 by any chance? :smiley:

I have a more direct question actually: is there any way to feed the task watchdog without using a delay? And also without disabling it entirely :expressionless:

This is the decoded backtrace of the crash that occurs specifically in the while loop with the .run() functions:

Decoding stack results
0x40088dd8: invoke_abort at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/panic.c line 155
0x40089005: abort at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/panic.c line 170
0x400dd5df: task_wdt_isr at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/task_wdt.c line 174
0x40081137: delayMicroseconds at /Users/JTB/Library/Arduino15/packages/esp32/hardware/esp32/1.0.2/cores/esp32/esp32-hal-misc.c line 157
0x400d4476: AccelStepper::step1(long) at /Users/JTB/Documents/Arduino/libraries/AccelStepper/AccelStepper.cpp line 391
0x400d4341: AccelStepper::step(long) at /Users/JTB/Documents/Arduino/libraries/AccelStepper/AccelStepper.cpp line 348
0x400d45c2: AccelStepper::runSpeed() at /Users/JTB/Documents/Arduino/libraries/AccelStepper/AccelStepper.cpp line 60
0x400d47c9: AccelStepper::run() at /Users/JTB/Documents/Arduino/libraries/AccelStepper/AccelStepper.cpp line 185
0x400d3110: MotorsTc(void*) at /Users/JTB/Downloads/Sterling-AC-1 :: Work/Sterling-AC-1/Sterling-AC-1.ino line 1801
0x40087925: vPortTaskWrapper at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c line 143

And this is the AccelStepper .cpp referenced in the backtrace: https://textuploader.com/1dxww

I have not encountered with the freeRTOS watchdog timer. I do not believe your issue is with the watchdog timer, the watchdog timer is working properly and triggering an error at the proper time.

You've got two functions that, when called, go off into outer space somewhere, whiles the functions are visiting, the timer goes off and causes a system reset. That's the proper operation.

Also, running freeRTOS taskings without proper traffic control will result in errors.

Have you considered searching for a ESP32 library to run your motors?

Idahowalker:
You've got two functions that, when called, go off into outer space somewhere, whiles the functions are visiting, the timer goes off and causes a system reset. That's the proper operation.

It does work fine with vtaskdelays in any loops it encounters in space, the thing I'd like to do most is just feed TWD without causing any delay in the accelstepper loop since it needs to run unhindered for smooth, fast stepping

stepper1.setCurrentPosition(0);

    stepper1.setMaxSpeed(MaxSpeed1);
    stepper1.setAcceleration(Accel1);

    stepper2.setMaxSpeed(MAX_SPD);
    stepper2.setAcceleration(MAX_XCL);
    stepper2.moveTo(Move2_A[i]);
    stepper2.runToPosition();
    
    stepper2.setMaxSpeed(MaxSpeed2[i]);
    stepper2.moveTo(Move2_B[i]);

Here, you do several instructions with one stepper lay it aside and then work on the other stepper. Why not intermingle, interlace, the code for 1 and 2?

I was able to work out a way around eventually, the guys at the ESP32 github suffered through it: Task Watchdog & AccelStepper · Issue #2892 · espressif/arduino-esp32 · GitHub

It actually turns out to be an issue with the AccelStepper step function using delayMicroseconds, which doesn't register with TWD. Any delay type that registers will make your motors very slow, so right now there's no easy way to feed it. With TWD disabled on the stepper core, you will still get reboots when you come to the stepper.run() function in a loop - there seems to be some timing issues with the task switching. So the solution for now is to disable TWD on on the stepper core and set that core's task to higher priority than the other.

And that's it for now, works fine. I think they're gonna keep working on it, but it's fine like this for now I suppose.

I ran into a similar issue where I need to loop very fast to drive some motion control hardware.

I did a bit of digging and found 2 solutions.

  1. esp_task_wdt_reset() which resets the Tast Watchdog Timer (TWDT) on behalf of the currently running task. This is the correct way of feeding the dog.
#include "esp_task_wdt.h"
esp_task_wdt_reset();

But, this method will not work if there are any other threads that have been registered with esp_task_wdt_add(), because it checks to make sure all threads have called esp_task_wdt_reset() recently, and only feeds the dog if so. If you have a thread that must run so fast it blocks the other threads that are on the same core, then you can

  1. write to the wdt registers directly:
#include "soc/timer_group_struct.h"
#include "soc/timer_group_reg.h"
void feedTheDog(){
  // feed dog 0
  TIMERG0.wdt_wprotect=TIMG_WDT_WKEY_VALUE; // write enable
  TIMERG0.wdt_feed=1;                       // feed dog
  TIMERG0.wdt_wprotect=0;                   // write protect
  // feed dog 1
  TIMERG1.wdt_wprotect=TIMG_WDT_WKEY_VALUE; // write enable
  TIMERG1.wdt_feed=1;                       // feed dog
  TIMERG1.wdt_wprotect=0;                   // write protect
}

This method is hella fast, but it is also bypasses and underminds the rtos's ability to catch other threads when they enter unstable states. So, it should not be used unless you are running a simple application, and don't care about blocking the other threads... which I am...

3 Likes

vaporizationator you helped to save my Day

#include "esp_task_wdt.h"
esp_task_wdt_reset();

was not working for me

but
void feedTheDog(){

was working thanks have a nice year.

1 Like