I would like to check why my code is constantly blocked once the additional functions are added. I am doing some simple gesture recognition on my Arduino Nano 33 BLE Sense board using two main libraries Arduino_LSM9DS1 and Arduino_TensorFlowLite. The code works perfectly for collecting data and doing inference but once some other function (even stupid print function to print some stupid sentence) is added the code is blocked and cannot be run. Can you please provide any insights? I am not sure if problem is related to software or maybe my hardware is broken.
Here is the code:
#include <Arduino_LSM9DS1.h>
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/tflite_bridge/micro_error_reporter.h>
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "model.h"
#include <string.h>
// Constants
const float accelerationThreshold = 2.5;
const int numSamples = 120;
int samplesRead = numSamples;
const char* GESTURES[] = {
"mov1",
"mov2",
"mov3"
};
#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))
int highestIndex = 0;
float* resultArray;
const char* finalResult;
float highestValue;
// Global variables for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;
tflite::AllOpsResolver tflOpsResolver;
const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;
constexpr int tensorArenaSize = 8 * 8192;
byte tensorArena[tensorArenaSize];
// Global variables for IMU
float aX, aY, aZ, gX, gY, gZ;
void printResult(){
Serial.println("This is the function that should print the final result!");
}
void collectData(){
while (samplesRead == numSamples) {
if (IMU.accelerationAvailable()) {
// read the acceleration data
//Serial.println("Read acceleration: ");
IMU.readAcceleration(aX, aY, aZ);
// sum up the absolutes
float aSum = fabs(aX) + fabs(aY) + fabs(aZ);
// check if it's above the threshold
if (aSum >= accelerationThreshold) {
// reset the sample read count
samplesRead = 0;
break;
}
}
}
// check if the all the required samples have been read since
// the last time the significant motion was detected
while (samplesRead < numSamples) {
//Serial.println("Samples read is lower than num of samples. ");
//Serial.print("Check accelerometer data: ");
// check if new acceleration AND gyroscope data is available
if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
// read the acceleration and gyroscope data
IMU.readAcceleration(aX, aY, aZ);
IMU.readGyroscope(gX, gY, gZ);
float minAccel = -2.0;
float maxAccel = 2.0;
float minGyro = -1000.0;
float maxGyro = 1000.0;
tflInputTensor->data.f[samplesRead * 6 + 0] = (2.0 * (aX - minAccel) / (maxAccel - minAccel)) - 1.0;
tflInputTensor->data.f[samplesRead * 6 + 1] = (2.0 * (aY - minAccel) / (maxAccel - minAccel)) - 1.0;
tflInputTensor->data.f[samplesRead * 6 + 2] = (2.0 * (aZ - minAccel) / (maxAccel - minAccel)) - 1.0;
tflInputTensor->data.f[samplesRead * 6 + 3] = (2.0 * (gX - minGyro) / (maxGyro - minGyro)) - 1.0;
tflInputTensor->data.f[samplesRead * 6 + 4] = (2.0 * (gY - minGyro) / (maxGyro - minGyro)) - 1.0;
tflInputTensor->data.f[samplesRead * 6 + 5] = (2.0 * (gZ - minGyro) / (maxGyro - minGyro)) - 1.0;
samplesRead++;
}
}
}
const char* runInference(){
TfLiteStatus invokeStatus = tflInterpreter->Invoke();
for (int i = 0; i < NUM_GESTURES; i++) {
resultArray[i] = tflOutputTensor->data.f[i];
}
highestValue = resultArray[0];
for (int i = 1; i < NUM_GESTURES; i++){
if (resultArray[i] > highestValue){
highestValue = resultArray[i];
highestIndex = i;
}
}
return GESTURES[highestIndex];
}
void setup(){
Serial.begin(9600);
while(!Serial);
IMU.begin();
tflModel = tflite::GetModel(model);
if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
Serial.println("Model schema mismatch!");
while (1);
}
else {
Serial.println("Model schema matches!");
}
tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize);
tflInterpreter->AllocateTensors();
Serial.print("Arena Used Bytes of the deployed model: ");
Serial.println(tflInterpreter->arena_used_bytes());
tflInputTensor = tflInterpreter->input(0);
tflOutputTensor = tflInterpreter->output(0);
Serial.println("Setup done!");
}
void loop(){
Serial.println("Welcome!");
collectData();
if (samplesRead == numSamples){
finalResult = runInference();
}
Serial.println("Final result is: ");
Serial.println(finalResult);
// when printResult function is commented it works without issues!
printResult();
}
No, I did not, but I read it now. I did not find too much information that can be useful in my case to be honest (or any solution they provided). Based on my insights, seems that any function that does not contain components from TensorFlowLite library somehow blocks the code execution. Is there maybe some bug in tensorflow lite library flow or something like that?
The problem always occurs when the tenserflow invoke() is called, but i'm not sure why. Also make sure that you are not blocking the loop function because otherwise the bluetooth will not work.
do you see the Serial.println("Setup done!");
which is at the end of the setup?
Initially you block in collectData, may be having a bool that would let you know when the buffer is full would be better than a blocking call?
It is not blocking the collectData when there is no other function that does not include tensorflow variables. In my case, I have one additional function printResults that blocks the code every time it is not commented and there is an initiative to call it once inference part is done. That's weird no
This may be a comprehension of Serial usage problem, but I don't have time to investigate your code, so you'll have to. Please read this carefully.
You have initialized Serial at 9600 baud. That means at most, 960 characters can be sent per second.
You have verbose print commands in your code, which I would applaud for debugging purposes, but...
Serial has a limited output buffer(can be 64 or 128 chars, Arduino type dependent). If you print characters faster than that buffer is being emptied, when it fills, your code will be blocked while space is freed in the transmit buffer, painfully, one character at a time.
For example, if you try to send 20 characters, but there's only room for 5, you will be blocked for 15 character times before the Serial Print routine returns to your code. The next call will likely block immediately, for almost the entire character time of the message you send. And, it will only improve from there on if your code goes silent until the buffer empties.
So, to test if this is in fact occurring in your code, increase the Serial baud rate to 115200, set Serial Monitor to match, and observe your system performance. If the problem goes away, that was it. If it takes much longer to appear, it's related to the problem, but you'll still have to reduce verbosity.
If there's no change, look elsewhere.
After I added this part, the code behaves as expected, executing functions by an order does not matter if they contain or not tensorflow lite components.
