Recording analog input data stream and playing back for development

Problem
I'm trying to write a program for my Arduino project to detect REM (the dreaming stage of sleep) with electrode sensors. However, it's time consuming to only test my code at night when I'm sleeping.

I'd like a way to record the electrode sleep data at night, and then be able play it back through the arduino program when I'm awake, or run sections of it, in order to develop the code.

Any guidance appreciated.

Details/ More Questions:

data file size
If I collect the data from the serial monitor and save it to a text file, the data ends up being around 3mb, because it's a stream of data for about 8 hours.

recording
It would be ideal to save the data without having to be connected to the computers' serial port for battery life reasons, but the serial monitor works for now.

playback latency?
The part that confuses me most is how would I play pack the data? The data is a stream, but Arduino doesn't run its loops at consistent intervals.

For example, I might collect this sleep data in 4 void loops:

millis: 1  electrode: 234
millis: 7  electrode: 238
millis: 14  electrode: 230
millis: 19  electrode: 229

But I cannot ensure that I play back the electrode data through the device at the exact millisecond intervals they were recorded because the arduino loops seem to vary in millis.

And to complicate matters more, if I had to add an SD card to read back the mock data, wouldn't reading the SD card slow the loops down further? These inconsistencies with playing back the recorded data seem like they would effect the my ability to write a program for real data based on mock playback data.

But am I over thinking this? Does it not make that much of a difference?

How might one go about recording and playing back long streams of data or thinking about this problem?

Update:
It seems like I might want to make my program work at a consistent sampling rate- sampling rate tutorial. Up until now, I've been taking data on every loop, instead of at a consistent rate. This would probably mean changing my existing code. One thing I'm still confused about is how do you know what sampling rate to pick to ensure that you don't have any void loops that take longer than the samping rate?

What sort of data are you recording? Is it sound? or an analog output of some sensor?

I'm recording from electrodes:

Each connected to the an analog pin on the Uno.

By definition, your data is NOT streaming. You are asking the A/D converter for an integer of data and then storing it. There is NO data until you ask for it. So, remember the time since the last reading in milliseconds and store the difference that along with the integer of data.
On playback, wait the stored number of milliseconds associated with that data and then do your thing with the data.

So, about 104 bytes per second. That shouldn't be too fast for an SD card.

Only one of each?

  1. Thanks for the clarification. Do you know what people are referring to to when they say data stream in arduino land?

  2. Wouldn't this method of waiting the number of milliseconds not work in the case where the void loop lakes longer than the interval of the stored data? The loops tend to vary on their own already. But even more, if I'm reading the data from an SD card, wouldn't that slow down the loop by quit a bit? So if the data's interval is 6 ms, but the void loop is now 20ms I wouldn't be able to play back the data in time.

But would reading from the SD card (or writing to it) on every loop slow the loops down and throw off the times? Is what I'm asking making sense?

Three electrodes each. But 3 electrodes = 1 flow of data. So one flow of data for EMG and one for EOG.

Not really. In my world, streaming data means you have NO control of when the data is sent, not even the interval between packets. You have to always be ready to read and process it.

The Sparkfun OpenLog is an inexpensive and reliable standalone data logger that simply records everything you choose the Serial.print(). Gigabytes of it, if you like.

If you have a 5V Arduino, be sure to use a level shifter on the OpenLog RX input.

So 6 analog inputs. 104 bytes per second would be about 17-1/3 bytes per analog input per second. Can you show a short sample of the currently logged (Serial Monitor) data?

Sorry for the confusion. 3 electrodes === 1 electrode channel. 1 electrode channel per 1 analog input. So 2 analog inputs.

Here's an example. This isn't the raw data though.

06:30:34.827 -> average: 34
06:30:35.336 -> average: 22
06:30:35.861 -> average: 35
06:30:36.351 -> average: 28
06:30:36.825 -> average: 35
06:30:37.339 -> average: 33
06:30:37.841 -> average: 35
06:30:37.876 -> EOG interpretted: LEFT
06:30:38.320 -> average: 40
06:30:38.830 -> average: 23
06:30:39.361 -> average: 41
06:30:39.895 -> average: 53
06:30:40.056 -> EOG interpretted: RIGHT

average means the average of the last buffer of raw EMG muscle data.
EOG interpretted is the programs calculation of the raw EOG eye movement data numbers into RIGHT or LEFT eye movements.

I could get a sample of the raw data to show you too in a bit.

Oh. So it is "Only one of each" kind of board.

It looks like your 'average' values are about once per half second. Did you want to record the RAW data or just the average every half second?

I sort or wanted to record the raw data, but maybe I don't need to. So I think I see what you're getting at- maybe if the data I'm recording doesn't need to be super high resolution I won't have issues with the loops not being fast enough.

Most microcontrollers ADC's have a free running mode. You can use that and from the interrupt place the sample in a buffer. At some interval you would then copy the buffer and persist it somewhere, sd card, eeprom, whatever.

Basically the reverse. Have a timer interrupt read samples from a buffer and output them using PWM or a DAC (whatever is available). While the interrupt routine does its thing you have some time to read stored samples into a second buffer. When the buffer the interrupt routine reads from is empty, you swap the two buffers. Now the interrupt has fresh data to read from and you have another buffer to read data into.

This technique is called double buffering if you want to read up on it.

thanks for teaching me about this. I'll read up on it

1 Like

Which presumably refers to a differential pair of two electrodes, and a third electrode as an "active ground".

For these microvolt signals, it is not just a matter of a single electrode picking up a signal.

Yes exactly

I created sample code for doing what I need to do. I haven't tried in the real environment yet, but wanted to share what I have so far.

@johnwasser's suggestions inspired my idea to create a consistent sampling rate and set it to 100ms. My scientist friends are collecting sleep data with consistent sampling rates, so this seems to be standard. I have yet to change my actual code to collect at a consistent sampling rate, so there may be new issues when I try this in the real code.

@nicolajna suggested some more advanced ideas I have yet to implement or understand fully. I couldn't find any simple examples of double buffering on arduino with simple int values. So I will need to look more into these suggestions if I run into more issues, which may happen when I try this in the real code/environment. If anyone knows of a good tutorial or example on double buffering, please share.

Run the test code
You can test this code with just an SD card reader connected. A0 will pick up electrical noise and give you some changing readings even with nothing attached to it. But if you want to record more meaningful data, connect some sensor or potentiometer to A0.

Before starting, set bool recording = to recording or playback mode.

Serial monitor in recording mode

15:29:06.404 -> Mode: RECORDING
15:29:06.404 -> 
15:29:06.404 -> Initializing SD card...initialization done.
15:29:06.437 -> deleting file.
15:29:06.437 -> DATA.TXT doesn't exist. Now creating....
15:29:06.475 -> 
15:29:06.475 -> RECORDING...
15:29:06.509 -> 100: 312
15:29:06.615 -> 200: 214
15:29:06.728 -> 300: 110
15:29:06.800 -> 400: 44
15:29:06.909 -> 500: 24
15:29:07.017 -> 600: 20

Serial monitor in playback mode

15:29:17.519 -> Mode: PLAYBACK
15:29:17.519 -> 
15:29:17.519 -> Initializing SD card...initialization done.
15:29:17.519 -> 
15:29:17.519 -> PLAYING BACK...
15:29:17.621 -> handling data: 100: 312
15:29:17.697 -> handling data: 200: 214
15:29:17.811 -> handling data: 300: 110
15:29:17.925 -> handling data: 400: 44
15:29:17.997 -> handling data: 500: 24
15:29:18.111 -> handling data: 600: 20

Full Code

/*
   SD card attached to SPI bus as follows for Arduino Uno:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 5
*/
#include <SPI.h>
#include <SD.h>

File myFile;

bool recording = // set whether we should be recording data or playing back data
//  false;
  true;

bool readData = true; // false when we saved a data point from the SDcard to the input string
// and haven't handled it yet during a sampling rate moment,
// so we to pause reading data, ie pause before reading the next data point from the SD card,
// until we handle the last data point. Not used in recording mode.
bool playData = true; // false when we get to the end of the SDcard file. Not used in recording mode.


int samplingDelta = 100;

unsigned long startTime = millis();
unsigned long actualTime;
unsigned long loops = 0;

int stringIndex = 0;
const size_t BUF_DIM = 50;
char inputString[BUF_DIM];
char inputChar;
int  reading;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.println();
  Serial.print("Mode: ");
  Serial.println(recording ? "RECORDING" : "PLAYBACK");

  Serial.println();

  Serial.print("Initializing SD card...");

  if (!SD.begin(5)) {
    Serial.println("initialization failed!");
    while (1);
  }

  Serial.println("initialization done.");

  if (recording) {
    deleteAndRecreateFile();

    myFile = SD.open("DATA.TXT", FILE_WRITE);

  }

  if (!recording) {
    myFile = SD.open("DATA.TXT");

  }

  Serial.println();
  Serial.println(recording ? "RECORDING..." : "PLAYING BACK...");



}


void loop() {
  actualTime = millis();

  if (!recording && readData && inputChar != EOF ) {
    inputChar = myFile.read();
    while (inputChar != '\n' && inputChar != EOF) {
      inputString[stringIndex] = inputChar;
      stringIndex++;
      inputChar = myFile.read();

    }
    if (inputChar == '\n' || inputChar == EOF) {
      readData = false;
    }
    stringIndex = 0;
  }
  if (recording) {
    reading = analogRead(A0);
  }

  bool timeToCollectData = actualTime - startTime >= samplingDelta;

  if ( timeToCollectData) {
    if (recording) {
      writeToSD(reading, actualTime,  myFile);
    } else {
      if (playData) {

        if (inputChar != EOF) {
          handleData(inputString);
          readData = true;

        }
        if (inputChar == EOF) {
          playData = false;
        }
      }

    }

    startTime = startTime + samplingDelta;
  }

  if (recording && millis() > samplingDelta * 15) stopCollectingData(); // To limit the recording for testing purposes.
}


void handleData(String data) {
  // real code would go here
  Serial.print("handling data: ");
  Serial.println(data);
}
void  writeToSD(int val, unsigned long time, File file) {

  if (file) {
    Serial.print(time);
    Serial.print(": ");
    Serial.println(val);

    file.print(time);
    file.print(": ");
    file.println(val);

  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }

}

void deleteAndRecreateFile() {
  Serial.println("deleting file.");

  SD.remove("DATA.TXT");

  if (SD.exists("DATA.TXT")) {
    Serial.println("---ERROR DELETING----DATA.TXT exists.");

  }
  else {
    Serial.println("DATA.TXT doesn't exist. Now creating....");
    myFile = SD.open("DATA.TXT", FILE_WRITE);
    myFile.close();
  }
}
void stopCollectingData() {
  myFile.close();
  Serial.println("Done recording. delaying code for a long time...");

  delay(9000000000);

}