Sensor readings slow down after a couple of minutes

Official Hardware Nano Family Nano 33 BLE
1

Hi all,

I'm currently experimenting with an Arduino Nano BLE Sense and a relatively high sampling rate (every 50ms) for the IMU, Pressure sensor and Thermometer.

I used the provided Arduino_* sensor libraries first, but switched to the Adafruit ones, because they seemed a bit more reliable.

With all the mentioned libraries, i noticed a strange behaviour:
At the beginning, everything works perfectly. After a couple of minutes (or seconds, depending on the used libraries), the sensor readings start to take more and more time, up to half a second. There is no error, they just take ages to complete.

Does this have something to do with a clogged I2C-Bus or something like that?
I already tried to increase the bus speed without success...

At the beginning, I was wondering, if it could be connected to BLE, but if i comment everything out, there is no difference in the behaviour.

If you start the sketch and look into the serial console, you'll see that the updates happen quite quick, but after a few minutes, the logs after "BEGIN Sensors::Loop()" take longer and longer...

Interestingly ALL actions on the Wire1 bus seem to take ages. Its not a single sensor, that behaves strange...

void Sensors::Loop(SystemState &state) {
  if(DEBUG) Serial.println("BEGIN Sensors::Loop()");
  float temperature, humidity;
  ReadTemperatureAndHumidity(temperature, humidity);
  float pressure = ReadPressure();
  float acc_x, acc_y, acc_z;
  ReadAcceleration(acc_x, acc_y, acc_z);
  float g_x, g_y, g_z;
  ReadGyroscope(g_x, g_y, g_z);
  float m_x, m_y, m_z;
  ReadMagneticField(m_x, m_y, m_z);
  DataPoint newItem = {state.VehicleState, millis(), pressure, temperature, acc_x, acc_y, acc_z, g_x, g_y, g_z, m_x, m_y, m_z};
  state.CurrentDataPoint = newItem;
  if(DEBUG) Serial.println("END Sensors::Loop()");
}

void Sensors::ReadAcceleration(float &acc_x, float &acc_y, float &acc_z ) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadAcceleration()");
  acc_x = -999, acc_y = -999, acc_z = -999;
  if (IMU.accelerationAvailable()) {
    IMU.readAcceleration(acc_x,acc_y,acc_z);
  }
  if(DEBUG) Serial.println("END Sensors::ReadAcceleration()");
}

void Sensors::ReadGyroscope(float &g_x, float &g_y, float &g_z) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadGyroscope()");
  g_x = -999, g_y = -999, g_z = -999;
  if (IMU.gyroscopeAvailable()) {
    IMU.readGyroscope(g_x,g_y,g_z);
  }
  if(DEBUG) Serial.println("END Sensors::ReadGyroscope()");
}

void Sensors::ReadMagneticField(float &m_x, float &m_y, float &m_z) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadMagneticField()");
  m_x = -999, m_y = -999, m_z = -999;
  if (IMU.gyroscopeAvailable()) {
    IMU.readMagneticField(m_x, m_y, m_z);
  }
  if(DEBUG) Serial.println("END Sensors::ReadMagneticField()");
}

float Sensors::ReadPressure() {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadPressure()");
  return lps35hw.readPressure();
}

void Sensors::ReadTemperatureAndHumidity(float& temp, float &humidity) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadTemperatureAndHumidity()");
  sensors_event_t temp_reading;
  sensors_event_t humidity_reading;
  hts.getEvent(&temp_reading, &humidity_reading);

  temp = temp_reading.temperature;
  humidity = humidity_reading.relative_humidity;
  if(DEBUG) Serial.println("END Sensors::ReadTemperatureAndHumidity()");
}

You can find the complete code here:

Thanks in advance :slight_smile:

Chris

2

What happens if you increase the serial line speed?

3

I'm already on Serial.begin(115200);

The problem is still there if I set DEBUG to false. (No Serial.println then)

4

Serial goes above 1Mbps, IIRC. That's quite a bit faster

You didn't mention that you'd tried turning off debug.

5

I optimized the code a bit and removed some testing stuff.

With DEBUG=False, I was now able to run the code for approx. 20 minutes until the problem occured. An update happened every 4 seconds then and another interesting symptom appeared:
The pressure sensor reported suddenly a pressure of -0.000244140625 hpa. (Before the problem appeared, it was around the normal 1004 hpa)

6

@anon73444976 This board has native USB and the baud rate in Serial.begin is ignored.

The Arduino Nano 33 BLE uses mbedOS. mbedOS is a bit different than the Arduino framework and that can cause issues. Here is an example of an issue I have seen in the past that might help you look into the right place.

In mbedOS there are objects created for many things and most official examples on the mbed web page create them globally and keep them forever e.g., a DigitalOut object for an output pin.

From what I understand the Arduino code often creates these object again and again. This can create all kinds of issues. e.g., when you use digitalWrite a DigitalOut object is created and then the value is written to the pin.
In earlier versions the creation of the object turned the pin into an input, then back into an output and then the value was set. This caused a short glitch in most cases and for some users it dropped the output voltage of the pin because the pin toggled between input and output continuously because they used digitalWrite in loop() with nothing else. I think this issue may be fixed now.

I suspect one of your functions causes an object to be created again and again and this causes some resource allocation in mbedOS that is not handled properly.

7

Hi Klaus and thank you for your answer.

If I understand you correctly, something like this would be problematic, right?

  DataPoint newItem = {state.VehicleState, millis(), pressure, temperature, acc_x, acc_y, acc_z, g_x, g_y, g_z, m_x, m_y, m_z};
  state.CurrentDataPoint = newItem;

And if the problem is in one of the used libraries, i'm not able to solve the problem except I fork the lib, right?

Is there actually a way to debug this?
I already ordered a J-LINK edu mini to try to get a bit more information...

I tried the memory info function from [SOLVED] Measure free SRAM of Nano 33 BLE Sense at runtime - #3 by arhyrhy123 and the memory footprint seams to be rock solid:

Thread: 0x2000AFDC, Stack size: 1088 / 32768
Thread: 0x20016238, Stack size: 504 / 4096
Thread: 0x20002A08, Stack size: 400 / 512
Thread: 0x200029C4, Stack size: 104 / 768
Thread: 0x20011EA0, Stack size: 128 / 256
Heap size: 22925 / 189096 bytes

This stays the same for the hole run. I think, if there would be a ressource allocation problem, we would see it there, right?

8

The problem also doesn't seem to be deterministic. Sometimes it slows down after 20 seconds, some other times it stays fast for 20 minutes.

I had the feeling, the behaviour changes if I play around with Wire1.setClock(); and that it slows down faster if I increase the clock speed, but I'm not sure.

At the moment I'm thinking, that maybe a flaky USB connection could be the issue as its so random...

UPDATE: nope - it has the same issue when run from another USB port or a LiPo

I now have a second IMU (MPU6050) attached on Wire0. When the slowdown happens, the MPU6050 still reads fast as normal. Only the devices attached to Wire1 are getting slow.

I also tried to comment out different internal sensors with their init and loop code. This doesn't change the behaviour. and I tried another BLE Sense, the problem is reproducable...

9

Not a solution for your problem, sorry, but a few remarks:

First I would try to find out, if it is a special function creating the problem. So I would test the loop only with one sensor (then the next, etc.).

Next I would check the timing of loop itself. Call micros() at the beginning and at the end and calculate the difference. By the way, who is calling loop? What about that timing?

For me the mbedOS is a very bad thing, it seems to destroy the timing of many libraries I've written in the past years. Fortunately you can bypass the OS with direct register access. So I have written my own BLE beacons without that big ArduinoBLE-library.
Currently I analyze the IMU, because the reading of acceleration values takes about a millisecond, which I think is a long time. Maybe I will learn, that it needs that time for the I2C, but then there should be an asynchronous routine with interrupt handling in the background.

Good luck ...
Edit: Ok, I have seen You made the proposed test while I was writing.

10

I read the sensors on Wire1 one by one now with an interval of 33ms between the reads and set a flag, if all reads are done.
This seems to make everything a bit more robust, but its still a game of luck.
Another interesting thing:
If I replace the Arduino_LSM9DS1 library with the Adafruit_LSM9DS1 one, the problem appears instantly.

I'm pretty sure, there is a major problem in the whole I2C-stuff...

UPDATE:
My work around is now to measure, how long the sensor reading takes and to reset the I2C bus, if it takes longer than expected. This works fine, but creates small gaps in the collected data.

11

There is a thing called HEAP fragmentation. I am not an expert on this stuff but this seems like a source of unpredictable timing and likely would not show up in the size information. If the HEAP algorithm needs to search for a block of memory that is big enough it could cause timing changes. Having multiple threads with requests of different sizes could make the behavior appear random.

Do you have a logic analyser? There are some for around $10 which you can use with an open-source software called PulseView.

  • Google: "24MHz 8channel logic analyzer"
  • https://sigrok.org/wiki/PulseView
    The software has build in decoders for all kinds of protocols e.g. I2C, SPI,... . You can use a digital pin as trigger and signal to show you the start and end of each individual request. This can show you where the time gets lost.
12

I really don't think, it has something to do with the heap... My "work around" at the moment is to reinitialize the I2C bus when I recognize, things start to take too long.
(By setting PIN_ENABLE_SENSORS_3V3 and PIN_ENABLE_I2C_PULLUP to LOW and then HIGH and initializing the sensors again after that.) The I2C bus recovers completly doing this and its fast again after that. But it takes again a random time interval after that until it gets slow again.

Restarting the sensor libraries should make the heap fragmentation worse if that is the problem...
I do not do dynamic allocations in my own code (at the moment).

hmm maybe that is the problem? Everything on the stack? But the memory footprint seemed ok...

Not at the moment, but I'll look for one without chinese delivery times to Germany...

How do you attach the probes to the Wire1 interface? It's not connected to the Arduino's pins...

13

I discussed the problem with a colleague and we both think, that for our application (periodically sending sensor data via a BLE beacon) it is the better way not to use any mbedOS routines. So in the next days we will try to create our own TWI-driver based on direct access to the registers of the nRF52840 (which has self running procedures for TWI). This should also solve our problem of the millisecond delay when reading the values.

If we experience the problem you detected, we will try to integrate a kind of reset in the communication at reasonable time points (without destroying data). Hopefully it is not a problem of the sensor chip.

By the way, we never use dynamic memory allocation in embedded systems. We avoid the new-operator when building instances of classes. May be sometime someone has a convincing argument for it, but till now it has not happened.

14

I extracted my sensor reading code and created a "small" single file Arduino sketch, if someone wants to try to reproduce the problem.

The RGB LED blinks green for every sensor reading.
The sketch runs for a couple of seconds or even minutes and then breaks. You can see that when the RGB-LED is no longer flickering green. Alternativly you can enable DEBUG and attach the serial console. At that point the sensor readings are extremly slow which you can see in the debug print statements

It doesn't matter which libraries I use for the sensors. Some break sooner, some later...
This happens on all of my BLE sense's (I got three) :frowning:

I'm not sure - Maybe I did something in the code horribly wrong as I'm a C/C++ beginner...
But I think this minimal sample should be OK. Is this a hardware bug then? or something in the framework?

#include <Adafruit_HTS221.h>
#include <Wire.h>
#include "Adafruit_LPS35HW.h"
#include <Arduino_LSM9DS1.h>
#include <Adafruit_Sensor.h>

#define DEBUG true


Adafruit_LPS35HW lps35hw;
Adafruit_HTS221 hts;
sensors_event_t a, m, g, temp, hum;

unsigned long previousMillis = 0;
#define SAVE_INTERVAL 50


void InitBarometer() {
  if (!lps35hw.begin_I2C(0x5C, &Wire1)) {
    if(DEBUG) Serial.println("Failed to initialize pressure sensor!");
    while (1) {}
  }
  lps35hw.setDataRate(LPS35HW_RATE_25_HZ);
}

void InitIMU() {
  if (!IMU.begin()) {
    if(DEBUG) Serial.println("Failed to initialize IMU!");
    IMU.end();
    InitIMU();
  }
}

void InitTemperatureAndHumidity() {
  if (!hts.begin_I2C(0x5F, &Wire1, 0)) {
    if(DEBUG) Serial.println("Failed to init temperature sensor!");
    while (1);
  }
  hts.setDataRate(HTS221_RATE_12_5_HZ);
}



void setup() {
  // put your setup code here, to run once:
  Serial.println("(re)starting sensors");
  pinMode(LED_RED, OUTPUT);
  pinMode(LED_GREEN, OUTPUT);
  digitalWrite(LED_RED, LOW);
  Wire1.begin();
  InitIMU();
  InitTemperatureAndHumidity();
  InitBarometer();
  digitalWrite(LED_RED, HIGH);
}

void loop() {
  unsigned long currentMillis = millis();
  if(currentMillis - previousMillis >= SAVE_INTERVAL) {
    digitalWrite(LED_GREEN, LOW);
    // put your main code here, to run repeatedly:
    float humidity, temperature;
    ReadTemperatureAndHumidity(temperature, humidity);

    float ax, ay, az, gx, gy, gz, mx, my, mz;
    ReadAcceleration(ax, ay, az);
    ReadGyroscope(gx, gy, gz);
    ReadMagneticField(mx, my, mz);
    float pressure = ReadPressure();

    digitalWrite(LED_GREEN, HIGH);
    previousMillis = currentMillis;
  }
}

void ReadAcceleration(float &acc_x, float &acc_y, float &acc_z ) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadAcceleration()");
  if (IMU.accelerationAvailable()) {
    IMU.readAcceleration(acc_x,acc_y,acc_z);
  }
  if(DEBUG) Serial.println("END Sensors::ReadAcceleration()");
}

void ReadGyroscope(float &g_x, float &g_y, float &g_z) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadGyroscope()");
  if (IMU.gyroscopeAvailable()) {
    IMU.readGyroscope(g_y, g_x, g_z);   // X and Y seem to be invert and swapped
    g_x *= -1;
    g_y *= -1;
  }
  if(DEBUG) Serial.println("END Sensors::ReadGyroscope()");
}

void ReadMagneticField(float &m_x, float &m_y, float &m_z) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadMagneticField()");
  if (IMU.magneticFieldAvailable()) {
    IMU.readMagneticField(m_x, m_y, m_z);
  }
  if(DEBUG) Serial.println("END Sensors::ReadMagneticField()");
}

float ReadPressure() {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadPressure()");
  return lps35hw.readPressure();
}

void ReadTemperatureAndHumidity(float& temperature, float &humidity) {
  if(DEBUG) Serial.println("BEGIN Sensors::ReadTemperatureAndHumidity()");
  hts.getEvent(&temp, &hum);

  temperature = temp.temperature;
  humidity = hum.relative_humidity;
  if(DEBUG) Serial.println("END Sensors::ReadTemperatureAndHumidity()");
}
15

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.