ESP32 board freezes randomly when rapid serial printing

Hi,
I have a custom ESP32-S2 board (schematics and PCB files here) with a build-in MPU6050 sensor. I want the board to continuously send me sensor values via Serial at the highest possible frequency. On my computer I want to save this Serial input to a txt file.

Here is my code that is running on the board:

#include "WiFi.h"
#include "time.h"
#include "Wire.h"
#include <Adafruit_MPU6050.h>

Adafruit_MPU6050 mpu;

const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

#include "Freenove_WS2812_Lib_for_ESP32.h"
#define LED_PIN 1
Freenove_ESP32_WS2812 led = Freenove_ESP32_WS2812(1, LED_PIN, 0, TYPE_GRB);

const char* ssid       = "SSID";
const char* password   = "PASSWORD";

long timer = 0;
struct tm timeinfo;
time_t now;

sensors_event_t a, g, temp;
String append;

void setLED(uint8_t r,uint8_t g,uint8_t b) {
  led.setLedColorData(0, r, g, b);
  led.show();
}

void setup() {
  // ******** LED for debugging *********
  led.begin();
  led.setBrightness(10);  
  setLED(60,60,0); // yellow
  // **************************

  Serial.begin(115200);

  // ********* Sync time *********
  //connect to WiFi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED);
  //init and get the time
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  if(!getLocalTime(&timeinfo)){
    Serial.println("Failed to obtain time");
    return;
  }
  //disconnect WiFi as it's no longer needed
  WiFi.disconnect(true);
  WiFi.mode(WIFI_OFF);
  // **************************
  
  // ********* mpu6050 *********
  Wire1.begin();
  if (!mpu.begin(0x68, &Wire1)) {
    Serial.println("MPU6050 Chip wurde nicht gefunden");
    return;
  }
  mpu.setAccelerometerRange(MPU6050_RANGE_8_G);
  mpu.setGyroRange(MPU6050_RANGE_500_DEG);
  // **************************

  setLED(0,60,0); // green
}

void loop() {
  if (!getLocalTime(&timeinfo)) {
    Serial.println("Failed to obtain time");
    return;
  }
  time(&now);

  mpu.getEvent(&a, &g, &temp);

  append = ""; 
  append += String(now)+","+String(millis()); 
  append += ",";
  append += String(a.acceleration.x, 6);
  append += ",";
  append += String(a.acceleration.y, 6);
  append += ",";
  append += String(a.acceleration.z, 6);
  append += "\n";

  setLED(60,0,0); // red
  Serial.print(append);
  delay(20); 
  setLED(0,60,0); // green
}

On my computer I have this python script running to read the serial input:

import serial

def readserial(baudrate):
    try:
        with serial.Serial(port='/dev/ttyACM0', baudrate=baudrate) as ser:
            for line in ser:
                line = line.decode('utf-8')
                if line:
                    print(line.rsplit('\n')[0])
    except:
        with serial.Serial(port='/dev/ttyACM1', baudrate=baudrate) as ser:
            for line in ser:
                line = line.decode('utf-8')
                if line:
                    print(line.rsplit('\n')[0])

if __name__ == '__main__':
    readserial(115200)

After a while (perhaps 5 minutes, but sometimes also after several hours) I will suddenly not receive any new serial lines and the led on the board will just be red, meaning that the board is apparently stuck at the Serial.print command. I have tried to put a delay of 20ms after the Serial.print, but it doesnt help. I also tried to simply print three random floats instead of reading from the sensor, but it will still get stuck.

I am not sure if this is perhaps an issue with my computer. I am running on ubuntu for that matter.

Any ideas are appreciated. Thank you for your time

you might get in an infinite loop here

do you see the "Failed to obtain time" message?


don't use Strings to built your output, this is just using memory and CPU for nothing, just print the data

  mpu.getEvent(&a, &g, &temp);
  Serial.print(now); Serial.write(',');
  Serial.print(millis());  Serial.write(',');
  Serial.print(a.acceleration.x, 6);  Serial.write(',');
  Serial.print(a.acceleration.y, 6);  Serial.write(',');
  Serial.println(a.acceleration.z, 6);

try 1 million baud (2 million works on my Mac) rather than 1125200

remember print becomes blocking once the output buffer is full, so ultimately this will become your throttling factor.

writing data in binary (may be with a start and/or end marker) would minimise the amount of data you send over Serial and leave it to Python to build the ASCII representation.

My experience is that I found best to use tasks in python to listen to the Serial port, I gave an example in this post

3 Likes

Seriously ?? By default GPIO 1 is the Serial TX pin, i am shocked that it works at all !

Hi,
Can you please post an image of your PCB please? In jpg.

Thanks.. Tom... :grinning: :+1: :coffee: :australia:

1 Like

Thank you!
I occasionally see the "Failed to obtain time", when I turn on the board, but I simply restart, when that happens. Thats not related to the freezing imo.

Increasing the baudrate is a good lead, Im trying that right now!

Ah, thats probably my bad. I meant to write that we use an ESP32-S2. Anyway GPIO1 is definitely LED for us as is shown in the schematic.

Well that does make a difference !

1 Like

He said so and pointed at the design in the original post…

Hi,
We need to see the PCB layout from the CAD you used to produce the PCB design.
Your CAD should be able to EXPORT an image of the actual tracks and component pads etc.

We need to see how and where you routed your tracks and track width.

Thanks.. Tom.. :grinning: :+1: :coffee: :australia:

1 Like

Hi @fullmoonish,

this might not directly solve your issue but is probably interesting for you:

/*
  Forum:https://forum.arduino.cc/t/esp32-board-freezes-randomly-when-rapid-serial-printing/1199915
  Wokwi: https://wokwi.com/projects/384003858849711105

   ESP32Time library (example sketch)
   https://github.com/fbiego/ESP32Time/blob/main/examples/esp32_time/esp32_time.ino

*/

#include "WiFi.h"
#include "time.h"
#include "Wire.h"
#include <Adafruit_MPU6050.h>
#include "ESP32Time.h"

//#define WOKWI

#ifdef WOKWI
// Wokwi specific data for SDA/SCL interface
// and access to simulated WiFi
constexpr byte SDA_1 {15};
constexpr byte SCL_1 {16};
constexpr unsigned long I2C_FREQ {400000};
TwoWire I2C_1 = TwoWire(0);
const char* ssid       = "Wokwi-GUEST";
const char* password   = "";
#else
// Usual declaration
const char* ssid       = "SSID";
const char* password   = "PASSWORD";
#endif

Adafruit_MPU6050 mpu;

const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

ESP32Time rtc(gmtOffset_sec);  // offset in seconds GMT+1

#include "Freenove_WS2812_Lib_for_ESP32.h"
#define LED_PIN 1
Freenove_ESP32_WS2812 led = Freenove_ESP32_WS2812(1, LED_PIN, 0, TYPE_GRB);


long timer = 0;
struct tm timeinfo;
time_t now;

sensors_event_t a, g, temp;

void setLED(uint8_t r, uint8_t g, uint8_t b) {
  led.setLedColorData(0, r, g, b);
  led.show();
}

void setup() {
  // ******** LED for debugging *********
  led.begin();
  led.setBrightness(10);
  setLED(60, 60, 0); // yellow
  // **************************

  Serial.begin(115200);

  Serial.println("Serial started");

  // ********* Sync time *********
  //connect to WiFi
  WiFi.begin(ssid, password, 6);
  Serial.println("WiFi begin done");
  while (WiFi.status() != WL_CONNECTED);
  //init and get the time
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  if (!getLocalTime(&timeinfo)) {
    Serial.println("Failed to obtain time");
    return;
  }
  //disconnect WiFi as it's no longer needed
  WiFi.disconnect(true);
  WiFi.mode(WIFI_OFF);

  Serial.println("WiFi off, time received");
  // **************************

  // ********* mpu6050 *********
#ifdef WOKWI
  I2C_1.begin(SDA_1, SCL_1, I2C_FREQ);
  if (!mpu.begin(0x68, &I2C_1)) {
#else
  Wire1.begin();
  if (!mpu.begin(0x68, &Wire1)) {
#endif
    Serial.println("MPU6050 Chip wurde nicht gefunden");
    return;
  }
  mpu.setAccelerometerRange(MPU6050_RANGE_8_G);
  mpu.setGyroRange(MPU6050_RANGE_500_DEG);
  // **************************

  setLED(0, 60, 0); // green
  refreshLocalTimeEverySeconds(0);  // Get the NTP time immediately
}

void loop() {
  refreshLocalTimeEverySeconds(10);  // Get the NTP time every 10 seconds only
  measureEverymSec(10);              // Measure every 10 ms
  heartBeat();                       //
}

void refreshLocalTimeEverySeconds(int seconds) {
  unsigned long mSeconds = seconds * 1000UL;
  static unsigned long lastTime = 0;
  if (millis() - lastTime >= mSeconds) {
    lastTime = millis();
    if (!getLocalTime(&timeinfo)) {
      Serial.println("Failed to obtain time");
    } else {
      Serial.println("************ Local Time refreshed! **************");
      rtc.setTimeStruct(timeinfo);
    }
  }
}

void measureEverymSec(unsigned long msecs) {
  static unsigned long lastMeasurement = 0;
  char message[120];
  if (millis() - lastMeasurement >= msecs) {
    lastMeasurement = millis();
    mpu.getEvent(&a, &g, &temp);
    Serial.printf("%20d, %20d, %2.6f, %2.6f, %2.6f\n",
                  rtc.getEpoch(),
                  lastMeasurement,
                  a.acceleration.x,
                  a.acceleration.y,
                  a.acceleration.z);
  }
}

void heartBeat() {
  static byte state = HIGH;
  static unsigned long lastChange = 0;
  if (millis() - lastChange > 500) {
    lastChange = millis();
    state = !state;
    if (state) {
      setLED(255, 0, 0);
    } else {
      setLED(0, 255, 0);
    }
  }
}

You can check it out on Wokwi: https://wokwi.com/projects/384003858849711105

Main changes:

  • There are some #ifdef statements that allow to use the sketch in Wokwi simulation (but are not compiled if "#define WOKWI" is not active)
  • I added the use of "ESPTime.h" that allows to use the ESP32 internal RTC. This way you can reduce the access to the NTP server and make your application more independent from this resource.
  • RTC is updated once at the end of setup() and then every 10 sec in loop() (which could be changed to e.g. 3600 sec)
  • The measurement is now in a separate function where you can control the time between two mesasurements in milliseconds. It is 10 ms at the moment.
  • The led changes between red and green in loop() as a heartbeat indicator in a fixed sequence with 500 msec interval.
  • The ESP supports the fantastic function printf() that provides the standard formatted printing in an easy way.

However, be aware that printing with a certain speed may lead to an overflow of the tx buffer!

At 115200 baud you can assume about 11520 characters per second (maximum).

If your message contains e.g. 80 characters incl. newline the maximum messages/second would be 11520/80 = 144 messages which is about 6.9 msec per message. So with less than 80 characters the 10 ms should be ok.

But just calculate this for intended maximum message length and allow for some tolerance. If the speed is not enough, you may consider to transmit raw binary data instead of ASCII messages!

Good luck!

2 Likes

It popped into my head to use multiples of 115200 like they do for the upload speed of an ESP32 within the IDE, you may get a bigger error margin that way. On an AVR it makes sense to use multiples of 250Kbps since they are usually running at 16MHz and a baudrate is set from the clock speed.

Thank you! Those suggestions really made it look much cleaner.
I integrated most of it into my code:

#include "WiFi.h"
#include "time.h"
#include "Wire.h"
#include <Adafruit_MPU6050.h>
#include "ESP32Time.h"

Adafruit_MPU6050 mpu;

const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

ESP32Time rtc(gmtOffset_sec);  // offset in seconds GMT+1

#include "Freenove_WS2812_Lib_for_ESP32.h"
#define LED_PIN 1
Freenove_ESP32_WS2812 led = Freenove_ESP32_WS2812(1, LED_PIN, 0, TYPE_GRB);

const char* ssid       = "SSID";
const char* password   = "PASSWORD";

long timer = 0;
struct tm timeinfo;
time_t now;

sensors_event_t a, g, temp;

void setLED(uint8_t r,uint8_t g,uint8_t b) {
  led.setLedColorData(0, r, g, b);
  led.show();
}


String append;
void setup() {
  // ******** LED for debugging *********
  led.begin();
  led.setBrightness(10);
  setLED(60, 60, 0); // yellow
  // **************************

  Serial.begin(921600);

  // ********* Sync time *********
  //connect to WiFi
  WiFi.begin(ssid, password, 6);
  while (WiFi.status() != WL_CONNECTED);
  //init and get the time
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  if (!getLocalTime(&timeinfo)) {
    Serial.println("Failed to obtain time");
    return;
  }
  //disconnect WiFi as it's no longer needed
  WiFi.disconnect(true);
  WiFi.mode(WIFI_OFF);
  // **************************
  
  // ********* mpu6050 *********
  Wire1.begin();
  if (!mpu.begin(0x68, &Wire1)) {
    Serial.println("MPU6050 Chip wurde nicht gefunden");
    return;
  }
  mpu.setAccelerometerRange(MPU6050_RANGE_8_G);
  mpu.setGyroRange(MPU6050_RANGE_500_DEG);
  // **************************
  setLED(0,60,0);

  refreshLocalTimeEverySeconds(0);  // Get the NTP time immediately
}

void loop() {
  refreshLocalTimeEverySeconds(10);  // Get the NTP time every 10 seconds only
  measureEverymSec(10);              // Measure every 10 ms
}

void refreshLocalTimeEverySeconds(int seconds) {
  unsigned long mSeconds = seconds * 1000UL;
  static unsigned long lastTime = 0;
  if (millis() - lastTime >= mSeconds) {
    lastTime = millis();
    if (!getLocalTime(&timeinfo)) {
      Serial.println("Failed to obtain time");
    } else {
      rtc.setTimeStruct(timeinfo);
    }
  }
}

void measureEverymSec(unsigned long msecs) {
  static unsigned long lastMeasurement = 0;
  char message[120];
  if (millis() - lastMeasurement >= msecs) {
    lastMeasurement = millis();
    mpu.getEvent(&a, &g, &temp);
    setLED(60,0,0);
    Serial.printf("%20d, %20d, %2.6f, %2.6f, %2.6f\n",
                  rtc.getEpoch()-3600,
                  lastMeasurement,
                  a.acceleration.x,
                  a.acceleration.y,
                  a.acceleration.z);
    setLED(0,60,0);
  }
}

I am saddened to say that increasing the baudrate did not help. I will try to adjust the python code according to the suggestions from Jackson.
Also at this moment I am not sure how to get the PCB layout, because someone else made that for us. I will report back, when I can get it.
Thank you for your time!

I really think that is still quite often considering that there is an RTC running as well. Once an hour might be better. It depends a bit where you get it from i guess. Most router provide a source as wel, but constantly pinging

may be a bit more than is appreciated.

Personally i ask once a day at around quarter to 2.

I also think that a call to the NTP server every 3600 s (1h) should be absolutely sufficient.

However the 10 sec rate should not cause any trouble, especially because the function taken from my sketch does not stop loop() even if an NTP call fails.

Agreed there, but if we discard the time related functions, what remains is setLED(); and the MPU object.
Now something must be interfering with something else, and i agree that the time related functions are most likely not the culprit, but the led library that use freertos, may interfere with something else. Have you tried using a different library. And to confirm you could disable the ntp. Mind you even the rtc may be interfering. Libraries from different sources may accidentally use the same resources, if there are multiple possible causes the only way to get there is by elimination. Live without the LED, and live without NTP and RTC, now does the problem persist ?

I'm with you.

Sometimes the other way around is also leading to success: Start with the minimum function and step by step add further functions.

  • Only do the measurement and send a minimum of data (e.g. plus millis() time)
  • Next: Adjust RTC once in setup and use RTC.epoch only in loop()
  • Next: Add led signal

BTW: The change of the led color around Serial.printf does only last for the time the controller needs to format the string and to fill the tx buffer. It is not directly related to the transmission as that's done in an interrupt routine. For that reason I suggested just a simple heart beat routine..

So is the transfer of Characters from the UARTs TX-buffer to the UARTs TX-FIFO, and the processor can only be in one ISR at a time. (I am saying that, but with 2 cores that may not be true, but they may be attempting to use the same core.

It is a tiny amount of data being sent to the LED though.

What I wanted to tell @fullmoonish is that switching the led color just around Serial.printf is not related to successful transmission.

It only shows for a very(!) short time that the printf statement has been performed.

The heart beat function as I suggested should serve the idea to have a visual clue that loop() is still running.

To check if there is enough space in the tx buffer one can use the function:

 int freeBytesInTXbuffer = Serial.availableForWrite();

see https://www.arduino.cc/reference/en/language/functions/communication/serial/availableforwrite/

So a (blocking) routine could look like this:

void measureEverymSec(unsigned long msecs) {
  static unsigned long lastMeasurement = 0;
  constexpr int msgLen {120};
  char message[msgLen];
  if (millis() - lastMeasurement >= msecs) {
    lastMeasurement = millis();
    mpu.getEvent(&a, &g, &temp);
    int cnt = snprintf(message, msgLen, "%20d, %20d, %2.6f, %2.6f, %2.6f\n",
                       rtc.getEpoch(),
                       lastMeasurement,
                       a.acceleration.x,
                       a.acceleration.y,
                       a.acceleration.z);
    // This blocks while not enough space in TX buffer
    while (Serial.availableForWrite() < cnt);
    Serial.print(message);
  }
}

or not blocking, but with data loss (which I would prefer):

void measureEverymSec(unsigned long msecs) {
  static unsigned long lastMeasurement = 0;
  constexpr int msgLen {120};
  char message[msgLen];
  if (millis() - lastMeasurement >= msecs) {
    lastMeasurement = millis();
    mpu.getEvent(&a, &g, &temp);
    int cnt = snprintf(message, msgLen, "%20d, %20d, %2.6f, %2.6f, %2.6f\n",
                       rtc.getEpoch(),
                       lastMeasurement,
                       a.acceleration.x,
                       a.acceleration.y,
                       a.acceleration.z);
    // This skips sending if not enough space in TX buffer
    if (Serial.availableForWrite() >= cnt) {
      Serial.print(message);
    }
  }
}
1 Like