Bluetooth gesture sensor

Using Arduino Project Guidance
1

Hi! I'm using 2 arduinos, one as a peripheral and one as a central to collect data from people in real time performing gestures. I have a tensorflow model I've made and uploaded, the recognition software is now being integrated with the bluetooth code, which is where I'm getting stuck. I'm trying to have the peripheral send the block of information to the central to be printed to the serial monitor after every "game" cycle, but I cant seem to figure out how to crack the sending part. The pairing has also been funky recently, even though I'm trying the bluetooth pairing code on its own (the bluetooth gesture example from the arduino website) and its working fine. Below is my code for the peripheral (I'm trying to get a system working for the central/peripheral deal right now, but hooking an arduino nano ble sense rev2 up to an external battery is proving to be a much harder feat than I would have imagined, so its a work in progress):

#include <ArduinoBLE.h>
#include <Arduino_BMI270_BMM150.h>
#include <TinyMLShield.h>
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <tensorflow/lite/version.h>

#include "model.h"

const float accelerationThreshold = 2.5; // Threshold of significant acceleration in G'
const float gyroscopeThreshold = 500.0;
const int numSamples = 119;
const int maxRounds = 20;
const int maxGestureCount = 12;
const int vibrationPin = 9;
const int ledPin = 13;

const char* deviceServiceUuid = "19b10000-e8f2-537e-4f6c-d104768a1214";
const char* deviceServiceCharacteristicUuid = "19b10001-e8f2-537e-4f6c-d104768a1214";

BLEService gestureService(deviceServiceUuid);
BLEByteCharacteristic gestureCharacteristic(deviceServiceCharacteristicUuid, BLERead | BLEWrite);

int samplesRead = numSamples;
int currentRound = 1;
int punchCount = 0;
int flexCount = 0;

unsigned long roundStartTime = 0;
unsigned long lastVibrationTime = 0;

struct GestureData {
  const char* gestureName;
  unsigned long gestureTime;
};

GestureData gestureData[maxRounds];
int currentRoundIndex = 0;

// Global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;
tflite::AllOpsResolver tflOpsResolver;
const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TFLM. The size may need to
// be adjusted based on the model you are using.
constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// Array to map gesture index to a name
const char* GESTURES[] = {
  "punch",
  "flex"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (!BLE.begin()) {
    Serial.println("- Starting Bluetooth® Low Energy module failed!");
    while (1);
  }

  BLE.setLocalName("Arduino Nano 33 BLE (Peripheral)");
  BLE.setAdvertisedService(gestureService);
  gestureService.addCharacteristic(gestureCharacteristic);
  BLE.addService(gestureService);
  gestureCharacteristic.writeValue(-1);
  BLE.advertise();

  if (!IMU.begin()) {
    Serial.println("- Failed to initialize IMU!");
    while (1);
  }

  Serial.println("Nano 33 BLE (Peripheral Device)");
  Serial.println(" ");

  pinMode(vibrationPin, OUTPUT);
  pinMode(ledPin, OUTPUT);

  // Initialize the IMU
  if (!IMU.begin()) {
    Serial.println("Failed to initialize IMU!");
    while (1);
  }

  // Print out the sample rates of the IMUs
  Serial.print("Accelerometer sample rate = ");
  Serial.print(IMU.accelerationSampleRate());
  Serial.println(" Hz");
  Serial.print("Gyroscope sample rate = ");
  Serial.print(IMU.gyroscopeSampleRate());
  Serial.println(" Hz");

  Serial.println();

  // Get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  startNewRound();
}

void startNewRound() {
  digitalWrite(ledPin, LOW);

  // Randomly select the gesture
  int gestureIndex = random(NUM_GESTURES);
  if (gestureIndex == 0) {
    punchCount++;
  } else {
    flexCount++;
  }

  gestureData[currentRoundIndex].gestureName = GESTURES[gestureIndex];
  gestureData[currentRoundIndex].gestureTime = 0;

  // Check if the gesture count exceeds the maximum
  if (punchCount > maxGestureCount || flexCount > maxGestureCount) {
    endGame();
    return;
  }

  // Print the current gesture
  Serial.print("Current Gesture: ");
  Serial.println(GESTURES[gestureIndex]);

  // Start the timer
  roundStartTime = millis();
  Serial.println("Timer Start");
}

void endGame() {
  digitalWrite(ledPin, HIGH);
  Serial.println("Game Over");

  String bulkData;
  for (int i = 0; i < currentRoundIndex; i++) {
    bulkData += "Round ";
    bulkData += i + 1;
    bulkData += ": Gesture: ";
    bulkData += gestureData[i].gestureName;
    bulkData += ", Time Taken: ";
    bulkData += gestureData[i].gestureTime / 1000;
    bulkData += " sec\n";
  }

  // Set the value of the characteristic
  const char* bulkDataStr = bulkData.c_str();
  gestureCharacteristic.write(bulkDataStr);

  // Print the bulk data to the serial monitor
  Serial.print(bulkData);
}


void loop() {
  float aX, aY, aZ, gX, gY, gZ;

  while (samplesRead == numSamples) {
    if (IMU.accelerationAvailable()) {
      // Read the acceleration data
      IMU.readAcceleration(aX, aY, aZ);

      // Sum up the absolute values
      float aSum = fabs(aX) + fabs(aY) + fabs(aZ);

      // Check if it's above the threshold
      if (aSum >= accelerationThreshold) {
        // Reset the sample read count
        samplesRead = 0;
        break;
      }
    }
  }

  while (samplesRead < numSamples) {
    // Check if new acceleration AND gyroscope data is available
    if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
      // Read the acceleration and gyroscope data
      IMU.readAcceleration(aX, aY, aZ);
      IMU.readGyroscope(gX, gY, gZ);

      // Normalize the IMU data between 0 and 1 and store it in the model's input tensor
      tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
      tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
      tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

      samplesRead++;

      if (samplesRead == numSamples) {
        // Run inference
        TfLiteStatus invokeStatus = tflInterpreter->Invoke();
        if (invokeStatus != kTfLiteOk) {
          Serial.println("Invoke failed!");
          while (1);
          return;
        }

        // Find the recognized gesture
        int recognizedGestureIndex = 0;
        float maxConfidence = tflOutputTensor->data.f[0];
        for (int i = 1; i < NUM_GESTURES; i++) {
          if (tflOutputTensor->data.f[i] > maxConfidence) {
            recognizedGestureIndex = i;
            maxConfidence = tflOutputTensor->data.f[i];
          }
        }

        // Check if the recognized gesture matches the current gesture
        if (recognizedGestureIndex == 0 && GESTURES[recognizedGestureIndex] == GESTURES[punchCount - 1]) {
          unsigned long gestureTime = millis() - roundStartTime;
          Serial.print("Correct Gesture: ");
          Serial.print(GESTURES[recognizedGestureIndex]);
          Serial.print(", Time Taken: ");
          Serial.print(gestureTime / 1000);
          Serial.println(" sec");
          gestureData[currentRoundIndex].gestureTime = gestureTime;

          delay(200);
          startNewRound();
          currentRoundIndex++;

        } else if (recognizedGestureIndex == 0 && GESTURES[recognizedGestureIndex] == GESTURES[flexCount - 1]) {
          unsigned long gestureTime = millis() - roundStartTime;
          Serial.print("Correct Gesture: ");
          Serial.print(GESTURES[recognizedGestureIndex]);
          Serial.print(", Time Taken: ");
          Serial.print(gestureTime / 1000);
          Serial.println(" sec");
          gestureData[currentRoundIndex].gestureTime = gestureTime;

          delay(200);
          startNewRound();
          currentRoundIndex++;
        }
      }
    }
  }
}

if anyone has any changes they can think for me to make, I'd really apprecite the help! The void endgame part is the one that keeps getting caught in the compiler, but if any other areas need changing just let me know!

2

"caught in the compiler"..... ?
Helpers not having Your hardware and library set will not be able to help You. You might have to wait for useful help.....

3 
BLEByteCharacteristic gestureCharacteristic(deviceServiceCharacteristicUuid, BLERead | BLEWrite);

The ByteCharacteristic is for a single byte value.

I'm trying to have the peripheral send the block of information to the central to be printed to the serial monitor after every "game" cycle

String bulkData;
  for (int i = 0; i < currentRoundIndex; i++) {
    bulkData += "Round ";
    bulkData += i + 1;
    bulkData += ": Gesture: ";
    bulkData += gestureData[i].gestureName;
    bulkData += ", Time Taken: ";
    bulkData += gestureData[i].gestureTime / 1000;
    bulkData += " sec\n";
  }

You can use the BLEStringCharacteristic to write this data for the central to read. To use the StringCharacteristic, you need a fixed size large enough for the maximum situation. How long is the bulkData String going to be?

If for example it might be 40 characters, you could use

//BLEByteCharacteristic gestureCharacteristic(deviceServiceCharacteristicUuid, BLERead | BLEWrite);
BLEStringCharacteristic gestureCharacteristic(deviceServiceCharacteristicUuid, BLERead | BLEWrite, 40 );

Sending a shorter String is not an issue.

To write the String value, you do not need to convert to a c string.

 //const char* bulkDataStr = bulkData.c_str();
  gestureCharacteristic.write(bulkData);

Writing these long Strings is somewhat contrary to the basics of BLE which is designed around short exchanges. You may want to consider converting the sketch to use the underlying struct and its size in bytes but that is adding complexity to what you have.

One other thought is that you should develop the peripheral and read the value you send with a standard app on a phone like Light Blue. When the peripheral is solid, you can develop the Central to read the value.

4

Okay that makes sense, I didnt know it had the max of 40 characters. I reduced it so it sends the data back after each completed gesture, and now im looking to have that data stored in the central arduino to be printed later on. thank you for your input

5

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.