Esp32 crashing using FreeRTOS tasks

[ems912]

Hi all,

I have been fighting a really annoying bug lately and I must be missing something important here. For context I have set up a esp32 Json websocket server running on core 1 in the Loop(); function and I created a task with the handle Sensors that runs on Core 0 and it updates the sensor data and runs a few timers. Everything was running fine before I added some sanity checks for the sensors but now even when I get rid of the checks Core 0 starts to panic and restart in a loop.

I have confined the problem down to the Task function SensorHandling() that runs on Core 0 but I still can't find the problem. Maybe someone else could take a crack.

[ems912]

Here Is the code snippet:

//======================================
// Sensor polling logic
//======================================
void SensorHandling(void * pvParameters) {
  for (;;) {
    xSemaphoreTake(Ticket, portMAX_DELAY);
    
    //======================================
    // Time Update
    //======================================
    DateTime now = rtc.now();
    char buf1[] = "DDD hh:mm";
    time1 = now.toString(buf1);
    char buf2[] = "MMM DD YY";
    time2 = now.toString(buf2);
    
    switch (now.dayOfTheWeek()) {
      case 0:
        CurrentdayLED = SunL;
        CurrentdayPump = SunW;
        break;
      case 1:
        CurrentdayLED = MonL;
        CurrentdayPump = MonW;
        break;
      case 2:
        CurrentdayLED = TuesL;
        CurrentdayPump = TuesW;
        break;
      case 3:
        CurrentdayLED = WedL;
        CurrentdayPump = WedW;
        break;
      case 4:
        CurrentdayLED = ThuL;
        CurrentdayPump = ThuW;
        break;
      case 5:
        CurrentdayLED = FriL;
        CurrentdayPump = FriW;
        break;
      case 6:
        CurrentdayLED = SatL;
        CurrentdayPump = SatW;
        break;
    }
    
    //======================================
    // Pull and format sensor data
    //======================================
    float temptemp = sht31.readTemperature();
    float humtemp = sht31.readHumidity();
    temptemp = tempFilter.filter(temptemp);
    temp = static_cast<int>((1.8 * temptemp) + 32);
    hum = static_cast<int>(humFilter.filter(humtemp));
    tempsoil1 = analogRead(36);
    tempsoil2 = analogRead(39);
    tempsoil3 = analogRead(34);
    tempsoil4 = analogRead(35);
    tempsoil1 = Soil1Filter.filter(tempsoil1);
    tempsoil2 = Soil2Filter.filter(tempsoil2);
    tempsoil3 = Soil3Filter.filter(tempsoil3);
    tempsoil4 = Soil4Filter.filter(tempsoil4);
    soil1 = map(tempsoil1, 1500, 3700, 100, 0);
    soil2 = map(tempsoil2, 1500, 3700, 100, 0);
    soil3 = map(tempsoil3, 1500, 3700, 100, 0);
    soil4 = map(tempsoil4, 1500, 3700, 100, 0);
    
    //======================================
    // Light control logic
    //======================================
    if (now.hour() >= 8) {
      if (now.hour() >= (CurrentdayLED + 8)) {
        ledcWrite(0, 0);
      }
      else if (RBState == 1) {
        ledcWrite(0, LedValue);
      }
      if (now.hour() >= (CurrentdayLED + 8)) {
        ledcWrite(1, 0);
      }
      else if (WState == 1) {
        ledcWrite(1, LedValue);
      }
    }
    
    //======================================
    // Time set logic
    //======================================
    if (TimesetFlag == 1) {
      Serial.print("Time SET!");
      rtc.adjust(DateTime(Y, Mth, D, H, Min, 0));
      SaveStates();
      delay(1000);
      ESP.restart();
    }
    
    //======================================
    // Temp/Hum heater timer
    //======================================
    unsigned long currentMillis3 = millis();
    if (currentMillis3 - previousMillis3 > interval3) {
      enableHeater = !enableHeater;
      sht31.heater(enableHeater);
      Serial.print("Heater Enabled State: ");
      if (sht31.isHeaterEnabled()) {
        Serial.println("ENABLED");
      }
      else {
        Serial.println("DISABLED");
      }
      previousMillis3 = currentMillis3;
    }
    
    //======================================
    // Page refresh timer
    //======================================
    unsigned long currentMillis2 = millis();
    if (currentMillis2 - previousMillis2 > interval2) {
      PageRefresh();
      previousMillis2 = currentMillis2;
    }
   
    //======================================
    //Peripheral Watchdog
    //======================================
    unsigned long currentMillis4 = millis();
    if (currentMillis4 - previousMillis4 > interval4) {

      if (! rtc.begin()) {
        Serial.println("Couldn't find RTC");
        WifiDisconnect();
        Serial2.print("page 8");
        SerialSend();
        Serial2.print("t0.txt=");
        Serial2.print("RTC");
        SerialSend();
        vTaskDelete(Sensors);
      }
      else if (! rtc.isrunning()) {
        Serial.println("RTC is NOT running, Setting to default time!");
        rtc.adjust(DateTime(Y, Mth, D, H, Min, 0));
      }

      if (! sht31.begin(0x44)) {
        Serial.println("Couldn't find SHT30");
        WifiDisconnect();
        Serial2.print("page 8");
        SerialSend();
        Serial2.print("t0.txt=");
        Serial2.print("SHT30");
        SerialSend();
        vTaskDelete(Sensors);
      }
      if ((soil1 > 100) || (soil2 > 100) || (soil3 > 100) || (soil4 > 100)) {
        Serial.println("Soil sensor has failed!");
        WifiDisconnect();
        Serial2.print("page 8");
        SerialSend();
        Serial2.print("t0.txt=");
        Serial2.print("SOIL100");
        SerialSend();
        vTaskDelete(Sensors);
      }
      else if ((soil1 < 0) || (soil2 < 0) || (soil3 < 0) || (soil4 < 0)) {
        Serial.println("Soil sensor has failed!");
        WifiDisconnect();
        Serial2.print("page 8");
        SerialSend();
        Serial2.print("t0.txt=");
        Serial2.print("SOIL0");
        SerialSend();
        vTaskDelete(Sensors);
      }
      previousMillis4 = currentMillis4;
    }
    
    xSemaphoreGive(Ticket);
    vTaskDelay(1);
  }
}

[ems912]

Update I was able to decode the backtrace:

Core 0 register dump:
PC : 0x400e8453 PS : 0x00060f30 A0 : 0x800e8501 A1 : 0x3ffb4e70
A2 : 0x00000000 A3 : 0x3ffb4ec9 A4 : 0x00000001 A5 : 0x00000000
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x00000014 A9 : 0x00000000
A10 : 0x3ffcb0e4 A11 : 0x80000001 A12 : 0x8008ecc4 A13 : 0x3ffcaef0
A14 : 0x00000003 A15 : 0x00060023 SAR : 0x00000000 EXCCAUSE: 0x0000001c
EXCVADDR: 0x0000000c LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace:0x400e8450:0x3ffb4e700x400e84fe:0x3ffb4e90 0x400e7ee8:0x3ffb4eb0 0x400d57be:0x3ffb4ef0

I got this as the output:

Decoding 5 results 0x400e8453: Adafruit_I2CDevice::write(unsigned char const, unsigned int, bool, unsigned char const, unsigned int)** at C:\Users\emsbb\OneDrive\Documents\Arduino\libraries\Adafruit_BusIO/Adafruit_I2CDevice.cpp line 80 0x400e8450: Adafruit_I2CDevice::write(unsigned char const, unsigned int, bool, unsigned char const, unsigned int)** at C:\Users\emsbb\OneDrive\Documents\Arduino\libraries\Adafruit_BusIO/Adafruit_I2CDevice.cpp line 79 0x400e84fe: Adafruit_I2CDevice::write_then_read(unsigned char const, unsigned int, unsigned char, unsigned int, bool)** at C:\Users\emsbb\OneDrive\Documents\Arduino\libraries\Adafruit_BusIO/Adafruit_I2CDevice.cpp line 221 0x400e7ee8: RTC_DS1307::now() at C:\Users\emsbb\OneDrive\Documents\Arduino\libraries\RTClib\src/RTC_DS1307.cpp line 58 0x400d57be: SensorHandling(void)* at C:\Users\emsbb\OneDrive\Documents\GitHub\Garden_Monitor\V2\ESP32\backend/backend.ino line 189

Looks like its something to do with Adafruits ds1307 library?

[tjaekel]

Just guesses:
I see, you use a semaphore: but you trigger "yourself": the semaphore you are waiting for - you set also at the end. This looks like an "endless loop".

I could imagine this:
it is a "recursion": the RTOS handles your thread. You fire the xSemaphoreGive(Ticket); and the RTOS will look if need to release a thread potentially waiting for this semaphore. And RTOS sees the "same" thread again, to release.

Even it is still running, possible, that RTOS launches the thread code again, like a nested call. It is recursive at the end: the same thread "calls" itself again, before it has "finished".
You eat up your local variables (the stack). It is like a nested call of the same function again before the previous one has finished (recursion).

The error you get might not be related to root cause, maybe you run out of stack (and overwrite other variables, resulting in a "strange" behavior).

Why do you trigger yourself? Technically your thread runs endless, in a full-speed loop.
What happens if you:

• get rid of this xSemaphoreTake(Ticket, portMAX_DELAY);
• and also this xSemaphoreGive(Ticket);
• it just "triggers itself" to keep running (never waiting), so, keep it as an endless loop (no need to wait, it would not wait anyway)
• use just this vTaskDelay(1); (or: osDelay(1); ) in order to give other threads a chance to be scheduled as well

I would imagine: the RTOS calls your thread unlimited times, as nested calls and you run out of local variable space (stack), corrupting other memories.
Triggering yourself with a semaphore is not needed (and actually "strange"): "you" are running already, no need to wait for "yourself".

Or: you delete yourself via vTaskDelete(Sensors); But you fire the xSemaphoreGive(Ticket); still at the end. Maybe, the RTOS cannot find anymore who is waiting for this Semaphore.
The RTOS looks a bit strange to me. Sorry.
But just guesses...

[Idahowalker]

When running code in loop() and using freeRTOS the code in loop() is not guaranteed to run.

@ems912

EXCVADDR: 0x0000000c

Means you got a task exceeding its stack space or something like an array is being accessed out of bounds.

Post the whole code.

[Idahowalker]

@ems912

xSemaphoreTake(Ticket, portMAX_DELAY);

Are you sure you want to use portMAX_DELAY when you are using millis() for timing?

Go to the Adafruit site. Open a ticket and post all the info you posted in this post. They will fix the library if it's a library issue.

Another thing to try is use an eariler version of the library.

[tjaekel]

Yeah, also my guess: a "recursion" (triggering yourself with semaphore)...

[tjaekel]

An endless loop with an osDelay(ms) or vTaskDelay(1); only should do the same job.
The "self-triggering" via semaphore looks to me suspicious.

[qubits-us]

@ems912

vTaskDelay(1);

how fast is this looping??

[qubits-us]

@ems912

if (! sht31.begin(0x44))

how often does this happen??
it does allot..

[qubits-us]

@ems912

if (! rtc.begin())

this one is probably killing you too, but can't find source to verify..
calling these begins, destroys, recreates and inits them..
doing that in a forever loop will give you grief..

[tjaekel]

vTaskDelay(1); should wait 1 ms. So, you would loop every 1 ms.

Personally, using RTOS I would change a bit the thread code:

• place all init, e.g. rtc.begin() before the for(;;): this entry of the thread is just executed once (on thread init and start),
  the thread keeps looping in the for-loop
• use a semaphore to release inside the for-loop (to have a blocking point), but set the semaphore from the "outside" (release by another thread), not releasing itself
• if the thread code should run in for-loop, all the time: no need for semaphore, instead use this
  vTaskDelay(1); (or osDelay(1)) : this blocks for 1 ms, it gives other threads a chance to be activated
• I would also place all variable definitions before the for-loop, not inside the for loop, or: create a block with curly brackets and define local inside the block, but the block will be finished, the local variables released before the thread loops back

Manuals about semaphores state:
"The correct use of a semaphore is for signaling from one task to another. A mutex is meant to be taken and released, always in that order, by each task that uses the shared resource it protects. By contrast, tasks that use semaphores either signal or wait—not both."
You do both in the same task, assuming this is an issue.