Saving sensor data with a buffer while also saving images

Projects General Guidance
1

I have an ESP-Wroom-32, an Arducam Mega 5mp, an SD reader, and a BME680. I want to write a program that measures every 0.1 seconds with the BME680, and saves each measurement onto the SD card into a csv file along with a timestamp. Every 5 seconds, I also want to take a picture and save it to the SD card. Even if I have the camera and the SD card on different SPI buses it seems that it cannot save the image and sensor data simultaneously, so while it's saving the image I want the sensor data to be saved in the ESP's buffer. When the image finished uploading, I want it to save all data from the buffer into the csv file and clear the buffer. Is this even possible? Are there better options? (Later I'm planning to add more sensors to this setup, but I want to make it work with this sensor first. I'm planning to add an MPU9250, an SGP30, a SAM-M10Q, and a Senseair Sunrise.)

This is my current code:

#include <SPI.h>
#include <SdFat.h>
#include "Arducam_Mega.h"
#include <Wire.h>
#include <Adafruit_BME680.h>

#define CAMERA_CS_PIN 5
#define SD_CS_PIN 15
#define SD_SCLK_PIN 14
#define SD_MISO_PIN 25
#define SD_MOSI_PIN 13

SPIClass *hspi = new SPIClass(HSPI);

SdFat SD;
Arducam_Mega myCamera(CAMERA_CS_PIN);
Adafruit_BME680 bme;

int imageCounter = 1;

struct SensorData {
  unsigned long timestamp;
  float temperature;
  float pressure;
  float humidity;
  float gasResistance;
};

const int BUFFER_SIZE = 50;
SensorData buffer[BUFFER_SIZE];
int bufferIndex = 0;

unsigned long lastMeasurementTime = 0;
unsigned long lastImageTime = 0;
const unsigned long measurementInterval = 100; // 0.1 seconds
const unsigned long imageInterval = 5000; // 5 seconds

void setup() {
  Serial.begin(115200);

  hspi->begin(SD_SCLK_PIN, SD_MISO_PIN, SD_MOSI_PIN, SD_CS_PIN);

  Serial.println("Initializing SD card");
  if (!SD.begin(SdSpiConfig(SD_CS_PIN, SHARED_SPI, SD_SCK_MHZ(10), hspi))) {
    Serial.println("SD card initialization failed");
    while (1);
  }
  Serial.println("SD card initialized");

  Serial.println("Initializing camera");
  if (myCamera.begin() != CAM_ERR_SUCCESS) {
    Serial.println("Camera initialization failed");
    while (1);
  }
  Serial.println("Camera initialized");

  Serial.println("Initializing BME680 sensor");
  if (!bme.begin(0x77)) {
    Serial.println("BME680 initialization failed");
    while (1);
  }
  Serial.println("BME680 initialized");
  bme.setTemperatureOversampling(BME680_OS_8X);
  bme.setHumidityOversampling(BME680_OS_2X);
  bme.setPressureOversampling(BME680_OS_4X);
  bme.setGasHeater(320, 150);
}

void loop() {
  unsigned long currentTime = millis();

  if (currentTime - lastMeasurementTime >= measurementInterval) {
    lastMeasurementTime = currentTime;
    if (bme.performReading()) {
      if (bufferIndex < BUFFER_SIZE) {
        buffer[bufferIndex++] = {
          currentTime,
          bme.temperature,
          bme.pressure / 100.0,
          bme.humidity,
          bme.gas_resistance
        };
      } else {
        Serial.println("Buffer overflow! Consider increasing the buffer size.");
      }
    } else {
      Serial.println("Failed to read BME680 sensor data.");
    }
  }

  if (currentTime - lastImageTime >= imageInterval) {
    lastImageTime = currentTime;
    saveImageToSD();
    saveBufferToSD();
  }
}

void saveImageToSD() {
  Serial.println("Capturing image...");
  if (myCamera.takePicture(CAM_IMAGE_MODE_HD, CAM_IMAGE_PIX_FMT_JPG) == CAM_ERR_SUCCESS) {
    uint32_t imageLength = myCamera.getTotalLength();
    if (imageLength == 0) {
      Serial.println("Error: Image length is 0. Skipping save.");
      return;
    }

    char filename[20];
    snprintf(filename, sizeof(filename), "/image%d.jpg", imageCounter++);

    File imageFile = SD.open(filename, O_RDWR | O_CREAT | O_TRUNC);
    if (!imageFile) {
      Serial.println("Failed to open file for writing.");
      return;
    }

    uint8_t buffer[128];
    while (imageLength > 0) {
      uint8_t bytesToRead = (imageLength > sizeof(buffer)) ? sizeof(buffer) : imageLength;
      myCamera.readBuff(buffer, bytesToRead);
      if (!imageFile.write(buffer, bytesToRead)) {
        Serial.println("Error writing to SD card!");
        imageFile.close();
        return;
      }
      imageLength -= bytesToRead;
    }

    imageFile.close();
    Serial.print("Image saved to SD card as ");
    Serial.println(filename);
  } else {
    Serial.println("Failed to capture image.");
  }
}

void saveBufferToSD() {
  File csvFile = SD.open("/data.csv", O_RDWR | O_CREAT | O_APPEND);
  if (!csvFile) {
    Serial.println("Failed to open CSV file.");
    return;
  }

  for (int i = 0; i < bufferIndex; i++) {
    csvFile.print(buffer[i].timestamp);
    csvFile.print(",");
    csvFile.print(buffer[i].temperature);
    csvFile.print(",");
    csvFile.print(buffer[i].pressure);
    csvFile.print(",");
    csvFile.print(buffer[i].humidity);
    csvFile.print(",");
    csvFile.println(buffer[i].gasResistance);
  }

  csvFile.close();

  bufferIndex = 0;
  Serial.println("Sensor data saved to CSV file.");
}

This code, when run, takes images every 5 seconds, but sensor data is not saved in the interval that the image is being saved onto the SD card. Also even when the image is not being saved the sensor doesn't save data every 0.1 seconds, but that could also be the SD card's limitation. Any help is appreciated!

2

You can dig down in the picture saving software, how it runs.
Waiting for a more knowing helper is another option.

3

What sensor functions are you using? The response time of the gas and humidity sensors are in the seconds range.

4

Please explain what that means, keeping in mind that the MCU can do only one thing at a time.

Note that opening an SD file, writing some data, and closing it again is an extremely slow, energy intensive and error-prone approach.

5

The way your code is currently written, you take an image and then write the entire thing to the SD card before the function returns so there is no way you could be capturing any sensor data. It would be worthwhile to take print millis() from time to time to see how long each chunck of your code is taking. Or print out currentTime each time through loop(). You might be surprised. It will give you a good sense of how long things are taking.

6

When testing the BME680 separately and outputting data every 100ms on the serial monitor it works without problems. Sorry, shouldve mentioned this in the main post.

7

Can it not save an image onto the SD card while temporarily storing sensor data in the buffer?

If so, is there an alternative?

8

You have the general idea, just duplicate this logic so you have one for each action. HOWEVER doing an Open on the SD card every time will kill your performance. Place that in setup, then based on number of writes or a ties periodically write append the card.

9

Of course it can. Please rephrase your question.

The following is a serious mistake for long term data collection. Open the file once in setup, and close it again when you are done collecting data.

  File csvFile = SD.open("/data.csv", O_RDWR | O_CREAT | O_APPEND);
  if (!csvFile) {
    Serial.println("Failed to open CSV file.");
    return;
  }

  for (int i = 0; i < bufferIndex; i++) {
    csvFile.print(buffer[i].timestamp);
    csvFile.print(",");
    csvFile.print(buffer[i].temperature);
    csvFile.print(",");
    csvFile.print(buffer[i].pressure);
    csvFile.print(",");
    csvFile.print(buffer[i].humidity);
    csvFile.print(",");
    csvFile.println(buffer[i].gasResistance);
  }

  csvFile.close();
10

Sure you can read it that fast but does the data mean anything? I bet the results don't change on the display that fast.

11

Reading the Bosch datasheet, with oversampling set to x1, the combined measurement time is 10.6 ms (excluding gas resistance, which is the only one over 100ms)

12

OK, I'll come out and say it. A 100ms read time is ridiculous. The environment will not change that fast, and even if it did the sensor wouldn't respond that fast. A more reasonable read rate would give you more time to do other things.

For example a river water quality system I worked on sampled in the hour range. The NOAA monitoring site we used to compare the initial version with sampled every 15 minutes.

13

This is a mini-satellite project. I want to have as many data samples as possible, during a 100-120s fall from 1km altitude. I will also transmit the data through radio, not only save them on the SD card.

But if it’s advisable, I can work with a lower save rate.

14

The pressure would change at a fast rate and the pressure sensor may be able to keep up. The temperature and humidity sensors would lag what is actually happening so their data would be less useful.

To keep track of the temperature you would need a sensor with low thermal mass which would change temperature quickly. Possibly something like a pin head sized bead thermistor.

15

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.