Sensor Sampling Throughput Optimization

Hello All,

I’m a bit of a novice punching above my weight, but I’m making slow progress. I’m working on a project to take a bunch of sensor data and run some AI/ML against it. I have a bunch of sensors plugged into an 33 Nano IoT and I’m generating data. I need to get this data, ultimately, to a Kafka cluster, but my immediate concern is dumping it out to MQTT. I picked up the IoT, because I wanted to ship everythign over wifi, but I quickly realized that I’d rather have a ton of sampling and simplicity than worry about setting up wifi.

High level architecture:
[sensors (accel, mag, gyro, and an analog fsr)] → [Arduino, Serial print] → [Python with serial package, MQTT write].

It’s pretty elegant, IMO, but my sampling is a bit slower than I would like… I have some ideas as to how to speed things up, but I’m looking for some guidance before I dive into any specific corner.

Ideas:

  • I’m printing ~20 Serial.print()'s. I could condense this into one Serial.print(String(1)+…+String(n)). Would this be worth a shot? Or would this be negligible…
  • Batching. I don’t really like the idea of batching, because the stream format of the serial lines work so nicely when dumping it into MQTT, but I could be convinced if this is the ideal way.
  • Is using the Serial port the entirely wrong way? Would wifi have been faster or something else…?

My Arduino code:

void loop(void)
{
  //define sensor events
  sensors_event_t aevent, mevent, gevent;
  //get accel, mag
  accelmag.getEvent(&aevent, &mevent);
  //get gyro
  gyro.getEvent(&gevent);
  
  /* Display the accel results (acceleration is measured in m/s^2) */
  Serial.print(aevent.acceleration.x, 4); Serial.print(",");
  Serial.print(aevent.acceleration.y, 4); Serial.print(",");
  Serial.print(aevent.acceleration.z, 4); Serial.print(",");

  /* Display the mag results (mag data is in uTesla) */
  Serial.print(mevent.magnetic.x, 1); Serial.print(",");
  Serial.print(mevent.magnetic.y, 1); Serial.print(",");
  Serial.print(mevent.magnetic.z, 1); Serial.print(",");
  
  /* Display the results (speed is measured in rad/s) */
  Serial.print(gevent.gyro.x); Serial.print(",");
  Serial.print(gevent.gyro.y); Serial.print(",");
  Serial.print(gevent.gyro.z); Serial.print(",");

  //fsr
  fsrReading = analogRead(fsrAnalogPin);
  Serial.print(fsrReading); 
  Serial.print(",");
  Serial.println();

}

And here is my python code on the other side, for good measure:

import serial
import json
import paho.mqtt.client as mqtt

ser = serial.Serial('COM3', baudrate = 9600, timeout = 1)

broker="127.0.0.1"
port=1883
client1= mqtt.Client("control1")                           #create client object
client1.connect(broker,port)

while 1:

	arduinoData = ser.readline().decode("utf-8").split(",")
	formatted = 0
	if len(arduinoData) < 9:
		break
	formatted = {"accel":[{"x":arduinoData[0],"y":arduinoData[1],"z":arduinoData[2]}],"mag":[{"x":arduinoData[3],"y":arduinoData[4],"z":arduinoData[5]}],"gyro":[{"x":arduinoData[6],"y":arduinoData[7],"z":arduinoData[8]}],"fsr":arduinoData[9]}
	forjson = json.dumps(formatted)

    #establish connection + publish
	ret= client1.publish("arduino/pen",forjson)

One concern I have for the batch plan is that my use case requires as real-time as possible. I just have a gut feeling that packing and unpacking my data will introduce too much of a delay over streaming the sensor data as efficiently as possible...

An impressive project. I see no real question, more a pre party mingle. What is the issue?

So I was working on this late last night and had these questions. I worked through printing everything in one single Serial.print and I'm seeing amazing results -- 10x the number of samples per second.

The code I pasted is dropping ~18 samples per second.

In a single line, with ~ 20 String() operations, I'm dropping ~170 samples per second. I would have been happy around 30-60, so this is great!

Here's the code for someone else to understand this later:

  Serial.println(String(aevent.acceleration.x, 2) + String(",") + String(aevent.acceleration.y, 2) + String(",") + String(aevent.acceleration.z, 2) + String(",") + String(mevent.magnetic.x, 1) + String(",") + String(mevent.magnetic.y, 1) + String(",") + String(mevent.magnetic.z, 1) + String(",") + String(gevent.gyro.x) + String(",") + String(gevent.gyro.y) + String(",") + String(gevent.gyro.z) + String(",") + String(fsrReading));

Not much to it!

Analyze the huge amount of data You got. Then modify Your code to output the amount of data You prefer.
Set a variable to the time, millis(), when You do a printout. Just before printing out use an if( currentmillis > (lastprintmillis + intervall)) to limit the output.

Yeah, that's a great idea. I was thinking about doing the selective sampling downstream, but limiting my output at the source would have huge benefite downstream (e.g. process everything the same and I could even do joins on the sensor data without much preprocessing.

Good. Try that and see if You feel more comfortable with the situation.

treats:
So I was working on this late last night and had these questions. I worked through printing everything in one single Serial.print and I'm seeing amazing results -- 10x the number of samples per second.

The code I pasted is dropping ~18 samples per second.

In a single line, with ~ 20 String() operations, I'm dropping ~170 samples per second. I would have been happy around 30-60, so this is great!

I refuse to believe that converting your data to Strings and concatenating them, can possibly be a reason for a speed up. You must have changed something else. If you want to help people out who pass by, you should repost the entire new sketch.

Using the String class is more likely a reason for slowing down things, rather than speeding up.

For this project I'd be looking at an ESP8266 or ESP32 processor, as they can do the MQTT communication by themselves. Especially the second one should give you a nice speed boost, considering it runs at 20 times the speed of a regular Arduino.

Yeah, I definitely made my board purchase without thinking too much about it. I’m actually surprised I haven’t hit a dead end yet. It will get me through my POC/MVP. I’ll check out the ESP32, but I may need to find something that will run a model, should my experimentation work.

aarg:
I refuse to believe that converting your data to Strings and concatenating them, can possibly be a reason for a speed up. You must have changed something else. If you want to help people out who pass by, you should repost the entire new sketch.

Anyway, the sped up code. I ran some more tests and was able to confirm my findings. The single println is way faster. I think from a processing perspective, the string conversion is probably more expensive, but from a bandwidth perspective, there seems to be a lot of overhead with writing 20 times vs 1 time.

Slow Results:
29 second sample period
527 samples
18.17 samples/sec

Fast Results:
28 second sample window
4748 samples
169 samples per second

“Slow” Code:

#include <Wire.h>
#include <Adafruit_FXOS8700.h>
#include <Adafruit_FXAS21002C.h>

//assign unique ID to these sensors
Adafruit_FXAS21002C gyro = Adafruit_FXAS21002C(0x0021002C);
Adafruit_FXOS8700 accelmag = Adafruit_FXOS8700(0x8700A, 0x8700B);

//fsr initialize
int fsrAnalogPin = 0; // FSR is connected to analog 0
int fsrReading;      // the analog reading from the FSR resistor divider

void displaySensorDetails(void)
{ 
  Serial.println("LOADING!");
}

void setup(void)
{
  Serial.begin(200000);
  /* Wait for the Serial Monitor */
  while (!Serial) {
    delay(1);
  }

  /* Initialise the sensor */
  if(!gyro.begin())
  {
    /* There was a problem detecting the FXAS21002C ... check your connections */
    Serial.println("Ooops, no FXAS21002C detected ... Check your wiring!");
    while(1);
  }

  /* Display some basic information on this sensor */
  displaySensorDetails();

  /* Initialise the sensor */
  if(!accelmag.begin(ACCEL_RANGE_4G))
  {
    /* There was a problem detecting the FXOS8700 ... check your connections */
    Serial.println("Ooops, no FXOS8700 detected ... Check your wiring!");
    while(1);
  }
  
  /* Display some basic information on this sensor */
  displaySensorDetails();
}


void loop(void)
{
  //define sensor events
  sensors_event_t aevent, mevent, gevent;
  //get accel, mag
  accelmag.getEvent(&aevent, &mevent);
  //get gyro
  gyro.getEvent(&gevent);
  //get fsr
  fsrReading = analogRead(fsrAnalogPin);
  
  /* This is the fast code
  Serial.println(String(aevent.acceleration.x, 2) + String(",") + String(aevent.acceleration.y, 2) + String(",") + String(aevent.acceleration.z, 2) + String(",") + String(mevent.magnetic.x, 1) + String(",") + String(mevent.magnetic.y, 1) + String(",") + String(mevent.magnetic.z, 1) + String(",") + String(gevent.gyro.x) + String(",") + String(gevent.gyro.y) + String(",") + String(gevent.gyro.z) + String(",") + String(fsrReading));
  */

  /* This is the slow code
    /* Display the accel results (acceleration is measured in m/s^2) */
  Serial.print(aevent.acceleration.x, 4); Serial.print(",");
  Serial.print(aevent.acceleration.y, 4); Serial.print(",");
  Serial.print(aevent.acceleration.z, 4); Serial.print(",");

  /* Display the mag results (mag data is in uTesla) */
  Serial.print(mevent.magnetic.x, 1); Serial.print(",");
  Serial.print(mevent.magnetic.y, 1); Serial.print(",");
  Serial.print(mevent.magnetic.z, 1); Serial.print(",");
  
  /* Display the results (speed is measured in rad/s) */
  Serial.print(gevent.gyro.x); Serial.print(",");
  Serial.print(gevent.gyro.y); Serial.print(",");
  Serial.print(gevent.gyro.z); Serial.print(",");

  //fsr
  fsrReading = analogRead(fsrAnalogPin);
  Serial.print(fsrReading); 
  Serial.print(",");
  Serial.println();
  //*/
}

“Fast” Code:

#include <Wire.h>
#include <Adafruit_FXOS8700.h>
#include <Adafruit_FXAS21002C.h>

//assign unique ID to these sensors
Adafruit_FXAS21002C gyro = Adafruit_FXAS21002C(0x0021002C);
Adafruit_FXOS8700 accelmag = Adafruit_FXOS8700(0x8700A, 0x8700B);

//fsr initialize
int fsrAnalogPin = 0; // FSR is connected to analog 0
int fsrReading;      // the analog reading from the FSR resistor divider

void displaySensorDetails(void)
{ 
  Serial.println("LOADING!");
}

void setup(void)
{
  Serial.begin(200000);
  /* Wait for the Serial Monitor */
  while (!Serial) {
    delay(1);
  }

  /* Initialise the sensor */
  if(!gyro.begin())
  {
    /* There was a problem detecting the FXAS21002C ... check your connections */
    Serial.println("Ooops, no FXAS21002C detected ... Check your wiring!");
    while(1);
  }

  /* Display some basic information on this sensor */
  displaySensorDetails();

  /* Initialise the sensor */
  if(!accelmag.begin(ACCEL_RANGE_4G))
  {
    /* There was a problem detecting the FXOS8700 ... check your connections */
    Serial.println("Ooops, no FXOS8700 detected ... Check your wiring!");
    while(1);
  }
  
  /* Display some basic information on this sensor */
  displaySensorDetails();
}


void loop(void)
{
  //define sensor events
  sensors_event_t aevent, mevent, gevent;
  //get accel, mag
  accelmag.getEvent(&aevent, &mevent);
  //get gyro
  gyro.getEvent(&gevent);
  //get fsr
  fsrReading = analogRead(fsrAnalogPin);
  
  /* This is the fast code */
  Serial.println(String(aevent.acceleration.x, 2) + String(",") + String(aevent.acceleration.y, 2) + String(",") + String(aevent.acceleration.z, 2) + String(",") + String(mevent.magnetic.x, 1) + String(",") + String(mevent.magnetic.y, 1) + String(",") + String(mevent.magnetic.z, 1) + String(",") + String(gevent.gyro.x) + String(",") + String(gevent.gyro.y) + String(",") + String(gevent.gyro.z) + String(",") + String(fsrReading));
  
//
//  /* This is the slow code
//    /* Display the accel results (acceleration is measured in m/s^2) */
//  Serial.print(aevent.acceleration.x, 4); Serial.print(",");
//  Serial.print(aevent.acceleration.y, 4); Serial.print(",");
//  Serial.print(aevent.acceleration.z, 4); Serial.print(",");
//
//  /* Display the mag results (mag data is in uTesla) */
//  Serial.print(mevent.magnetic.x, 1); Serial.print(",");
//  Serial.print(mevent.magnetic.y, 1); Serial.print(",");
//  Serial.print(mevent.magnetic.z, 1); Serial.print(",");
//  
//  /* Display the results (speed is measured in rad/s) */
//  Serial.print(gevent.gyro.x); Serial.print(",");
//  Serial.print(gevent.gyro.y); Serial.print(",");
//  Serial.print(gevent.gyro.z); Serial.print(",");
//
//  //fsr
//  fsrReading = analogRead(fsrAnalogPin);
//  Serial.print(fsrReading); 
//  Serial.print(",");
//  Serial.println();
  
}