Seeking Higher Sampling Rates

So, I'm trying to monitor the oscillations of a wing structure with dynamic loading. I don't have a specified target sampling rate, however, it is my understanding that the ~70 Hz that I'm currently at will not cut it (100Hz would be better, 1000Hz is probably what I need). I'm using a single analog device and writing the value to an SD card. The sensor device must be standalone so I cannot remove the writing to the SD card from the process. I have tried manipulating several publicly available codes (that honestly are beyond my understanding) without any success. If anyone has a fool-proof solution to my problem of increasing the sampling rate, it would be greatly appreciated. I'm using an arduino Uno for reference.

I understand that I may be cutting corners trying to use the arduino as a data acquisition system, DAS, but I'm reluctant to start shopping for an alternative industrial DAS. I do believe it is possible, and no one has said I cannot achieve higher sampling rates, but with my novice understanding and skill, I have not been successful. Thank you all in advance.

I'm using a single analog device and writing the value to an SD card.

Is your "sample rate" problem one of reading the sensor, or logging the data to the card? What is the sensor?

Please post the code which is getting you to ~70Hz.

I believe the issue is simply a slow reading of the analog values at a higher frequency as my serial monitor outputs are approximately the same as rates when the data is logged to an SD card. The SD card logging does not seem to be the bottleneck at the moment, but that is not to say it won't be later. I'm using an RTC for stamping the data with the date, but I omit that from the loop for the sake of streamlining the code. As for the sensor, it is a simple opamp for a strain gauge. Not much info on it. The sampling rate is the same regardless of whether I'm using the analog input from this strain gauge or my airspeed sensor, for example.

Here is the code that I am using:

//SD Card Setup
#include <SD.h>
const int CS_PIN =10;
const int POW_PIN =8;

//Strain Gauge Setup
int analogPin = 3; // connected to analog pin 3
// outside leads to ground and +5V
int val = 0; // variable to store the value read

//RTC setup
// Date and time functions using a DS3231 RTC connected via I2C and Wire lib
#include <Wire.h>
#include "RTClib.h"

RTC_DS3231 rtc;

char daysOfTheWeek[7][12] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};

void setup () {

Serial.println("Initializing Card");
pinMode(CS_PIN, OUTPUT);
pinMode(POW_PIN, OUTPUT);
digitalWrite(POW_PIN, HIGH);
if (!SD.begin(CS_PIN))
{
Serial.println("Card Failure");
return;
}
Serial.println("Card Ready");

#ifndef ESP8266
while (!Serial); // for Leonardo/Micro/Zero
#endif

Serial.begin(230400);

delay(3000);

if (! rtc.begin()) {
Serial.println("Couldn't find RTC");
while (1);
}

if (rtc.lostPower()) {
Serial.println("RTC lost power, lets set the time!");
// following line sets the RTC to the date & time this sketch was compiled
rtc.adjust(DateTime(F(DATE), F(TIME)));
// This line sets the RTC with an explicit date & time, for example to set
// January 21, 2014 at 3am you would call:
// rtc.adjust(DateTime(2014, 1, 21, 3, 0, 0));
}
//RTC sample bits
DateTime now = rtc.now();

File dataFile = SD.open("Strain1.csv", FILE_WRITE);
if (dataFile)
{
dataFile.print(now.year(), DEC);
dataFile.print(", ");
dataFile.print(now.month(), DEC);
dataFile.print(", ");
dataFile.print(now.day(), DEC);
dataFile.print(", ");
dataFile.print(now.hour(), DEC);
dataFile.print(", ");
dataFile.print(now.minute(), DEC);
dataFile.print(", ");
dataFile.println(now.second(), DEC);
dataFile.close();
}
}

void loop () {

File dataFile = SD.open("Strain1.csv", FILE_WRITE);
if (dataFile)
{
dataFile.print(analogRead(analogPin));
dataFile.print(", ");
dataFile.println(millis());
dataFile.close();
}
}

I've tried using arrays as well to write the data to the SD card in larger packets, but I did not have success with that. That is the most accessible way that I can think of to overcome this issue, however, at this stage, my ability to write code is entirely based off stitching together from other bits of code available and deleting sections here and there. It has served me well for the time being, but it really limits my creativity and resources to tackle this alone. Thank you for your help, and let me know if there is any more specifics that I can provide you with to help me with this matter. Thanks, again.

Opening and closing the file for each write is hopelessly inefficient.

Keep it open.

There are several steps you can take.

First, change the SD library you use from SD.h which comes with the IDE to Sd.Fat. It definitely benchmarks as faster.

Second, as Mark T says keep the file open. There is a 512 byte buffer in the library where data is written before being written to the card with file.close() or file.flush(). SdFat (and maybe SD.h as well?) will automatically write the buffer when it is full. These block writes are actually quite fast compared to your 1000Hz sample rate. You may not see any significant gaps in your data. See this thread Explanation of SD write() and flush() please? - Storage - Arduino Forum

If you do wind up with breaks in the data collection, the programming gets more complex when you try to fill and empty multiple buffers. A mega is better than a uno for doing that as it has more memory.

Third, you can reduce the amount of data stored by writing numbers as binary bytes instead of the ascii characters. That involves using file.write instead of file.print. The resultant file will not be the easily read .csv human readible you are used to, and will require the use of a some sort of binary file reader. The tools for read/write of buffered binary data are found in some examples with SdFat ("AnalogBinLogger" and "Low Latency Logger") but again, the situation is getting more complicated.

There are things that can be done on the analogRead() to speed that up. See Gammon Forum : Electronics : Microprocessors : ADC conversion on the Arduino (analogRead)

Thank you both so much for your help. I'll research these suggestions and let you know how it goes. Thanks, again!

Alright. So, I went through and tested some of your recommendations, but had no success. I tried testing each strategy individually so I could keep track on any improvements. Anyway, I began with the Gammon Forum : Electronics : Microprocessors : ADC conversion on the Arduino (analogRead) link and added the following after void setup() in the original code above. I saw no change in sampling despite uncommenting several of these individually.

ADCSRA &= ~(bit (ADPS0) | bit (ADPS1) | bit (ADPS2)); // clear prescaler bits

// uncomment as required

// ADCSRA |= bit (ADPS0); // 2
// ADCSRA |= bit (ADPS1); // 4
// ADCSRA |= bit (ADPS0) | bit (ADPS1); // 8
// ADCSRA |= bit (ADPS2); // 16
// ADCSRA |= bit (ADPS0) | bit (ADPS2); // 32
// ADCSRA |= bit (ADPS1) | bit (ADPS2); // 64
// ADCSRA |= bit (ADPS0) | bit (ADPS1) | bit (ADPS2); // 128

Second, I tried using SdFat.h instead of SD.h, but this has not worked. It is suggested that all that has to be done is to replace "SD.h" with "#include <SdFat.h> & SdFat SD;", but I'm having issues. When I run the example datalogger sketch it returns a "error: file.open
SD errorCode: 0X13,0XFF". When I incorporate it SdFat.h into my original sketch (above) I receive an error regarding "'File' not declared in this scope" in "File dataFile = SD.open("Strain1.csv", FILE_WRITE);
if (dataFile)". I can see that these two scripts operate differently, but it is not apparent to me on how to stitch SdFat into my own code at this point. Certainly when the example sketch is not working. Oh, and regarding the SD card error code, it does write perfectly fine in my other sketches so I don't believe it is a wiring issue.

And, regarding not closing the file at the end of each script, is it as simple as omitting that command and having SdFat do the rest (once I get it to work), because otherwise it does not write?
I haven't worked with the filewrite yet since I have been focusing on the other methods first. I'll let you know how that goes.

I apologize in advance if you guys are really having to spell it out for me, but I am grateful for your help, patience and suggestions so far.

I'm not sure why you are having trouble migrating to SdFat. There is definitely some different syntax, and I would adapt to that. Here is your sketch written for SdFat without file closure.

I am seeing over 1300 reads in the one second test. I have added some additional time monitoring, and every 32 records or so, there is a slight delay in the readings.

//SD Card Setup
//#include <SD.h>

#include <SPI.h>
#include <SdFat.h>
SdFat sd;
SdFile dataFile;

const int CS_PIN  = 10;
const int POW_PIN = 8;

//Strain Gauge Setup
int analogPin = 3;     // connected to analog pin 3
// outside leads to ground and +5V
int val = 0;           // variable to store the value read

//RTC setup
// Date and time functions using a DS3231 RTC connected via I2C and Wire lib
#include <Wire.h>
#include "RTClib.h"

RTC_DS3231 rtc;

char daysOfTheWeek[7][12] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};


void setup () {
  Serial.begin(230400);
  Serial.println("Type any Character to begin");
  while (Serial.read() <= 0) { }

  Serial.println("Initializing Card");
  pinMode(CS_PIN, OUTPUT);
  pinMode(POW_PIN, OUTPUT);
  digitalWrite(POW_PIN, HIGH);
  if (!sd.begin(CS_PIN))
  {
    Serial.println("Card Failure");
    return;
  }
  Serial.println("Card Ready");

#ifndef ESP8266
  while (!Serial); // for Leonardo/Micro/Zero
#endif



  //delay(3000);

  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }

  if (rtc.lostPower()) {
    Serial.println("RTC lost power, lets set the time!");
    // following line sets the RTC to the date & time this sketch was compiled
    rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
    // This line sets the RTC with an explicit date & time, for example to set
    // January 21, 2014 at 3am you would call:
    // rtc.adjust(DateTime(2014, 1, 21, 3, 0, 0));
  }
  //RTC sample bits
  DateTime now = rtc.now();

  if (dataFile.open("Strain1.csv", O_RDWR | O_CREAT | O_AT_END)) //SdFat syntax
  {
    Serial.println("data file open");
    dataFile.print(now.year(), DEC);
    dataFile.print(", ");
    dataFile.print(now.month(), DEC);
    dataFile.print(", ");
    dataFile.print(now.day(), DEC);
    dataFile.print(", ");
    dataFile.print(now.hour(), DEC);
    dataFile.print(", ");
    dataFile.print(now.minute(), DEC);
    dataFile.print(", ");
    dataFile.println(now.second(), DEC);

    // dataFile.close();//leave file open
  }
  else
    Serial.println("error opening file");
}

void loop () {

  static unsigned long lastWrite = micros();
  dataFile.print(analogRead(analogPin));
  dataFile.print(", ");
  dataFile.print(millis());
  dataFile.print(", ");
  dataFile.println(micros() - lastWrite);
  lastWrite = micros();

  static unsigned long startLog = millis();
  if ((millis() - startLog) > 1000) {
    dataFile.close();
    Serial.println("close file--test finished");
    while (1); //stops sketch
  }
}

Cattledog, thank you! A huge thank you. Exactly what I needed, not what I deserved. I've played around with the code above and you've genuinely launched me on my way to completing the rest of my project. Thank you for your patience! I follow the code, but my writing experience is strictly limited to juggling arrays and playing with transfer functions with scripts in MATLAB. I'm learning lots, but every couple steps forward was a step backward in terms of new concepts and applications between the software and hardware. I do not consider my work on this "done", and I will keep researching the suggestions you gave before. Again, and again, thank you!

So, I've been messing around with the code and I can see the impact of closing the file to write concerning write speeds. That alone can drop the sampling back down to ~70Hz. However, I am scratching my head as to how to log data continuously. The code stores a 1000 samples and then writes them to the SD card. The writing seems to be the most time intensive event here. How can I get continuous logging without unnecessarily introducing the open() and close() required for writing in the loop?

I once worked on a system that had a high speed datalogger in it that might be applicable to what you're trying to do.

In it, there were two micros (I can't recall if they were MCUs or MPUs, but it doesn't really matter.) The high power micro did all of the actual "work," while the second, lower powered micro handled the permanent storage.

Between the two was a shared sRAM bus. I remember the original designer saying he'd set it to trigger an interrupt on the second controller to dump the data from the sRAM to the storage (which, in this case, was an old MMC.)

Perhaps you could work out an arrangement something like that.

However, I am scratching my head as to how to log data continuously.
How can I get continuous logging without unnecessarily introducing the open() and close() required for writing in the loop?

I think you just need to log data continuously. The library will take care of the block writes. You only need to actually close the file before removing the card. You may want to have some periodic file.close(), in case of a power disruption or hardware glitch, as without the file management housekeeping done with the file.close() statement you can't see the data which was written to the card.

You can trigger the file.close() with a time period (like in the sketch provided), with a number of samples count, with an external button press, or with Serial input ("Press any key to stop logging") if you test for it in loop.

For how long do you want to collect data, and ideally, how would you like to start and stop.

That sounds like a reasonable strategy Chazzy. I see how allocating the tasks between two separate microcontrollers may be the route for a more capable data acquisition system. However, as much as I would like to jump in and tackle that issue (and believe me, I would if I had more time), but I'm really not capable of delivering such a setup in a reasonable time. I have already had to rely on folks like yourself and colliedog to help me along this far.

And, colliedog, I follow what you're saying now regarding the stops and writes. In order to understand this all a little better, I included the memoryfree library to monitor memory usage as I grabbed data and set the number of samples up to 150000, simply for experimentation. I'm not sure if I'm actually looking at the correct memory that the analog values are stored, but I saw no drop in memory usage (probably looking in the wrong place). What is a concern though, and may be related to the stored values, is the sampling rate (after I modified it slightly) went from ~635 Hz to 430 Hz by the end of my 150000 sample test. I can live with that drop off, but it would be nice to avoid.

Ideally, I would like to log for ~20 minutes and will probably integrate in a push button for stopping and starting later. I believe I can do that easily enough, but I wanted to give you a greater context for what I'm looking for.

I'm not sure if I'm actually looking at the correct memory that the analog values are stored, but I saw no drop in memory usage (probably looking in the wrong place).

The analog values, time values, etc. are stored in a 512 buffer managed by the SdFat library. Values are placed in the buffer when read, and then removed when written to the card. There should be not change in memory use during the sketch because the 512 bytes are initially allocated, and it doesn't matter if they contain values or not.

What is a concern though, and may be related to the stored values, is the sampling rate (after I modified it slightly) went from ~635 Hz to 430 Hz by the end of my 150000 sample test. I can live with that drop off, but it would be nice to avoid.

I don't understand this. The time taken to write the 512 byte block to the card sometimes varies as the internal controller of the memory card does some file housekeeping. I have never used huge SD files, so I don' know if there is something about appending to the end of a long file which slows things down monotonically.

Have you sorted out all the issues between data acquisition and the writing of the data? Are you sampling at fixed intervals, or just letting the data acquisition free run and log the time? How are you measuring the sample rate, and what is the pattern of the decline?

Please post the code which shows your current "data chunk" and the benchmarking.

So, I don't have an explanation yet for you, but I will get back to you on that shortly (may be some time early next week, regretfully). When I ran the sketch on a shorter number of samples, I don't recall specifically how many, but I did see the aforementioned variation in sample rates. In order to get a larger sample, I ran 1.5 million samples and averaged 510 Hz over the course of < 1 hour. I do still see variations in sampling ranging from 626 to 480 at a quick glance.

For reference, I am simply counting the number of cells for a given whole second in excel. I'll plot these results (Hz. vs time during the 1.5 million sample grab) out for you when I have it complete. I believe, now only looking at a hand full of samples, the frequency steadily drops off with time and plateaus. I am assuming the sample rate holds at ~450 Hz as I saw a similar range in a much shorter sample that was about 2 minutes.

I'll let you know what I find. At the very least, I have a much stronger data acquisition device and a better understanding than I did before. Thank you, again for your time and help on this.

As mentioned before, I'll let you know what happens to the sampling rate and perhaps more importantly the sensitivity of the readings with time. Have a good weekend!

Alright, so I now have the results from the sampling rates. I let the previously posted sketch go for roughly 1500 seconds. During this time the sampling rates varied, but in a consistent manner.

0 - 100 seconds - ~581Hz
101-1000 seconds - ~526 Hz
1001-1500 seconds - ~484 Hz

The sampling plateaued at these rates with only slight variation, and the rate changed distinctly at the designated time intervals above.

The time between samples was very consistent. Interestingly, it appeared that every 20 samples the time between readings was twice as long. And, every 12060 (if I remember correctly) samples, there was a delay roughly 10 times the average time between readings.

I still haven't looked at the sensitivity of the actual values recorded, but I didn't notice anything initially. I'll post anything else I notice.

I have two observations. I changed the sketch slightly to include a record number because I need to read the large files in a text editor and not a spread sheet. I also took out the RTC code to make it easier for anyone else to try and benchmark.

First, I can see that the record speed changes at places where the record number or the millis() time record increases by a decade and there are more digits stored. So, I think there must be some overhead to filling the buffer or running the loop. I was reading the 3.3v supply so my analogRead() values were all three digits.

Second, when I was at record number 100,000 and millis() over 100,000 I was seeing speeds of approximately 850 records/second. This is in your 100 seconds range, and my rates are higher, even when having the record number in the field.

What kind of SD card are you using? I took data with both a class 6 full sized card and a class 4 mico card in an adaptor and was in the 850 HZ range on both. I was expecting the class 6 card to be better, but it wasn't in the file area I examined.

//SD.h 
//#include <SPI.h>
//#include <SD.h>
//File dataFile;

//SdFat.h 
#include <SPI.h>
#include <SdFat.h>
SdFat sd;
SdFile dataFile;

const byte CS_PIN  = 4;
const byte analogPin = A3;   
int val = 0; 

void setup () {
  Serial.begin(115200);
  Serial.println("Type any Character to begin");
  while (Serial.read() <= 0) { }

  Serial.println("Initializing Card");
  pinMode(CS_PIN, OUTPUT);
 
  if (!sd.begin(CS_PIN))//SdFat.h syntax
  //if(!SD.begin(CS_PIN))//SD.h syntax
  {
    Serial.println("Card Failure");
    while(1);
  }
  Serial.println("Card Ready");

  if (dataFile.open("ConstantWriteTest3.csv", O_RDWR | O_CREAT | O_AT_END)) //SdFat syntax
   
   // dataFile = SD.open("ConstantWriteTest.csv",FILE_WRITE);//SD.h
   // if(dataFile)//Sd.h
  {
    Serial.println("data file open");
  }
  else
  {
    Serial.println("error opening file");
    while(1);
  }
}

void loop () {
  static unsigned long recordNumber=0;
  static unsigned long lastWrite = micros();
  dataFile.print(recordNumber++);
  dataFile.print(",");
  dataFile.print(analogRead(analogPin));
  dataFile.print(",");
  dataFile.print(millis());
  dataFile.print(",");
  dataFile.println(micros() - lastWrite);
  lastWrite = micros();

  static unsigned long startLog = millis();
  if ((millis() - startLog) > 200000ul) {
    dataFile.close();
    Serial.println("close file--test finished");
    while (1); //stops sketch
  }
}

Interesting. I'm using a speed class 10 SD card by the way. Is there a chance that it could be the SD module itself? I am using the Phantom YoYo SD shields available through Amazon. I have another SD card reader and I will maybe give that a shot. I agree with you that the additional digit must be the issue with the plateaus in sampling rates. Good call on spotting that.

Is there a chance that it could be the SD module itself?

Yes, it's possible.

I am using the V3.0 of the ITEAD studios shield Stackable SD Card shield V3.0 - ITEAD Wiki

V3.0 contains an SPI signal level shifting IC. It looks to me the like the V2.0 shield sold through Yo Yo may have a resistor network rather than the IC. This is known to be slower, and there have been several threads on the forum about problems with the low cost resistor network modules like those sold by LC studios.

Most of the micro card modules have the level shifting IC.

Confirmed! By simply switching over from the Yoyo SD card module to the Adafruit manufactured module, sampling rates improved significantly using the exact same code as before.

0 -<10 seconds --800 Hz
10 -<100 seconds - ~744 Hz
100 -<1000 seconds - ~704 Hz

Thank you for the suggestion. I have yet to read up on the resistor networks you mentioned before, but I will soon!