Go Down

Topic: Aquisition rate with SD Card (Read 3318 times) previous topic - next topic

diogotec

Hello!

In a project I'm designing, I need to make aquisition of three sensors with a frequency of 333.3333... Hz (delays of 3 ms in 1 second), display it in the serial monitor and save it to and SD card.

My question is if saving data to SD card(it would be one println per each aquisition of the three sensors) makes the aquisition rate being changed? It is really important to me to have a minimum of 300Hz!

Thanks :)

The code I wrote is the following:
Quote
//aquisition variables
const int EMG_pin = A0;  // Analog input pin that the EMG signal is attached to
long aq_delay = 3;    // Signal aquisition rate;
unsigned long last_aq_time = 0;
int EMG_value;

//SD Card libraries & variables
#include <SPI.h>
#include <SD.h>
File myFile;


void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);

  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open("aquisition.txt", FILE_WRITE);


}

void loop() {
  // put your main code here, to run repeatedly:
  if(millis() - last_aq_time >= aq_delay){ //establishes the aquisition rate
    last_aq_time = millis();
    //signal aquisition
    EMG_value = analogRead(EMG_pin);

    //data display
    Serial.println(EMG_value);
    //data saving
    if (myFile) {
      myFile.println(EMG_value);
      // close the file:
      myFile.close();
    } else {
      // if the file didn't open, print an error:
      Serial.println("error opening test.txt");
    }

  }
}

Nick_Pyner

make aquisition of three sensors with a frequency of 333.3333... Hz (delays of 3 ms in 1 second), display it in the serial monitor ...........
I think you should re-think that, it might help with your SD recording problem.

diogotec

What do I need to rethink exactly?
The aqusition frequency has to be higher than 300Hz (according to Nyquist theorem).

DrDooom


Maybe you can read my post in another thread.

I think the direct writing on sd card's without buffering is awkward.

I can only report from the reading, but the writing will be similar / even more difficult.
If you write over several blocks (512 byte), you usually have long waiting times (~ 2-3 ms).
In another post, I have created a continuous reading process. This can read a 512 byte block in 1.05ms. However always a complete 512 byte block must be read.

If you want to write the data to a sd card and this over a longer period (and a buffer in the arduino ram would not be enough), I would definitely put on a continuous writing process! Possibly, it is enough to preallocate the target file.

Nick_Pyner

What do I need to rethink exactly?
The aqusition frequency has to be higher than 300Hz (according to Nyquist theorem).
You need to rethink the wisdom of sending data to a serial monitor @ 300 Hz. I don't know who the hell Nyquist is, but I bet he can't read a serial monitor updating data at 300Hz. I'm also betting you can't either. Nobody would have the faintest idea of what you are trying to do, or why, but it would appear that your whole design approach is suss. If you re-thought the sanity of the data to the monitor, you might get Arduino to turn out something useful thereto, and you might then have an easier time feeding that same data to the SD. 

diogotec

@Nick_Pyner, eletromyography signal (signal from muscles) requires such aquisition rate..............

diogotec

#6
Jul 20, 2017, 03:55 pm Last Edit: Jul 20, 2017, 04:01 pm by diogotec
@DrDoom thanks for your reply!
Sorry, but I still have questions.

1) About the buffer you are right! But I didn't understanded the 512 byte part (you said it takes longer but you recomend to do it?). How can I do that?
I want to write the time + 3 sensors in one line like this:
HOUR 1022     242    4234
Is it possible with what you are saying?

Nick_Pyner

It's not for me to dispute that, I have never heard of eletromyography. I'm talking about the wisdom of sending the data to a monitor 300 times a second, which is very likely just a dog-chasing-car situation, even if it is possible. Feel free to explain to me how good an idea it is because, right now, I think it is a seriously stupid one. Indeed, I would go as far as to say that at this stage, the only redeeming feature of the project being designed that is evident, is the USB cable connecting Arduino to serial monitor. This cable offers the opportunity to connect Arduino to a proper terminal, and thereby record the data direct to the PC hard drive, hence making the SD card redundant.  Maybe DrDoom has some pertinent comment but, along with my bets about your ability to read the serial monitor, I'm also willing to bet the aquisition rate of a PC hard drive will be somewhat better than that of an SD card. This does not necessarily solve your problem, it may be simply moving it, and you may have to get your microprocessor to do some microprocessing - before you send it anywhere..

diogotec

It's not for me to dispute that, I have never heard of eletromyography. I'm talking about the wisdom of sending the data to a monitor 300 times a second, which is very likely just a dog-chasing-car situation, even if it is possible. Feel free to explain to me how good an idea it is because, right now, I think it is a seriously stupid one. Indeed, I would go as far as to say that at this stage, the only redeeming feature of the project being designed that is evident, is the USB cable connecting Arduino to serial monitor. This cable offers the opportunity to connect Arduino to a proper terminal, and thereby record the data direct to the PC hard drive, hence making the SD card redundant.  Maybe DrDoom has some pertinent comment but, along with my bets about your ability to read the serial monitor, I'm also willing to bet the aquisition rate of a PC hard drive will be somewhat better than that of an SD card. This does not necessarily solve your problem, it may be simply moving it, and you may have to get your microprocessor to do some microprocessing - before you send it anywhere..
Signals that come from the body usually have high frequencies. For example, for recording the heart beat u need at least to record it at 100Hz.
The reason I want to record my data to an SD card is because I want to make my system portable.

Nick_Pyner

#9
Jul 20, 2017, 05:27 pm Last Edit: Jul 20, 2017, 06:20 pm by Nick_Pyner
Signals that come from the body usually have high frequencies.
I'm afraid that this makes not the slightest difference to what I said.
Quote
display it in the serial monitor and save it to and SD card.
and
Quote
The reason I want to record my data to an SD card is because I want to make my system portable.
Good luck with that. And somehow I thought you wanted to display it on the serial monitor. Don't forget to disconnect it.

diogotec

#10
Jul 20, 2017, 06:35 pm Last Edit: Jul 21, 2017, 01:14 am by diogotec
I have also seen this post: http://forum.arduino.cc/index.php?topic=49649.msg354701#msg354701
But I don't know how to apply it to my case, and it seems a bit old, I don't know if there are new methods.

Used the following code to test how much time it takes 19 ms to aquire data from the 3 sensores, print it to the serial and to the SD card (see code!) using O_CREAT | O_WRITE.


Quote
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);

  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open("asd.txt", O_CREAT | O_WRITE);


}

void loop() {
  // put your main code here, to run repeatedly:
  if(millis() - last_aq_time >= aq_delay){ //establishes the aquisition rate
    last_aq_time = millis();
    //signal aquisition
    line = analogRead(EMG_pin) + tabe + analogRead(EMG_pin) + tabe + analogRead(EMG_pin);

    //data display
    Serial.println(line);
    Serial.println(millis());
    //data saving
    if (myFile) {
      myFile.println(line);
      Serial.println(millis());
    } else {
      // if the file didn't open, print an error:
      Serial.println("error opening test.txt");
    }

  }

}
[\QUOTE]

diogotec

#11
Jul 21, 2017, 12:01 am Last Edit: Jul 21, 2017, 03:06 am by diogotec
Also, I don't know how I can store the three sensor values in a buffer! I know nothing about buffers..
Please help :x

DrDooom

Ok, first let us know more about your requirements.

So far we know that you want to store 3 sensor values ​​with a frequency of at least 300 Hz. Show + save. Even if ur first code only shows 1 sensor pin/value (const int EMG_pin = A0;).

OK. But how long will the values ​​be processed? 1 second? 10 seconds? 1 minute? 10 minutes or 1 hour? Is the measurement limited to a fixed duration or is the duration variable? Do you want to use your project professionally or is it a demo for the school / university? That explains how much time and work you want to invest. All this can lead to a possibly different implementation.

A buffer is nothing but a memory that temporarily stores data before it is processed or stored. So if you get your sensor values, they can either be displayed / saved directly, or you can only save it first. 100 values ​​before you display / save them.
Like this e.g.:
Code: [Select]

const int sensorBufferSize = 100;
uint16_t  sensorBuffer[sensorBufferSize][3];//2 Dimensional Array -> Buffer with 100x3 Values
int       sensorBufferIndex = 0;//where to store next sensor values in the buffer?
...
      if(sensorBufferIndex < sensorBufferSize){
        sensorBuffer[sensorBufferIndex][0] = analogRead(PIN_EMG_1);
        sensorBuffer[sensorBufferIndex][1] = analogRead(PIN_EMG_2);
        sensorBuffer[sensorBufferIndex][2] = analogRead(PIN_EMG_3);
        sensorBufferIndex++;
      } else {
        //TODO: Warning, Buffer Overflow!!! the buffer with 100 different values is filled up, cant store more, need to write to sd card for example!
      }

So u can hold some values bevor writing them. I think u should read this thread from this post. I have checked the performance for reading an sd card. writing is similar.

Quote
But I didn't understanded the 512 byte part
In the thread / post i have linked, i tested to read less and more than 512 byte.

The results explain the behavior.
A sd card contains many small 512 byte blocks. A file is distributed to many of these blocks. The loading / creation of a block takes time.
In order to avoid the problem as far as possible, one can use continuous reading and writing. The time for reading / creating blocks is thereby greatly minimized. After all, you have maximum <3ms time to write the data.
I guess you can write the first 512 byte without problems. After the first 512 byte, you will have to wait a while for the writing process.

diogotec

Thanks for your reply and help DrDooom!

Quote
But how long will the values ​​be processed? 1 second? 10 seconds? 1 minute? 10 minutes or 1 hour? Is the measurement limited to a fixed duration or is the duration variable? Do you want to use your project professionally or is it a demo for the school / university? That explains how much time and work you want to invest. All this can lead to a possibly different implementation.
The objective of my project is to create a professional portable device (this explains why I want to save it to SD card), that can be used 24h+ without stopping (duration variable!).

Quote
So u can hold some values bevor writing them. I think u should read this thread from this post. I have checked the performance for reading an sd card. writing is similar.
So, according to your tests, using the SDfat Bibliothek with the File Class and a buffer improves speed, right?

The code you show is a buffer containing 100x3 elements, but how can I save only 512 bytes?

DrDooom

#14
Jul 21, 2017, 01:48 pm Last Edit: Jul 21, 2017, 02:07 pm by DrDooom
Well, that was just a simple example to buffer data.
Data buffering can ensure that writing to a sd card does not delay your 300 Hz interval.
If you want to save data permanently and over a long period, the buffer will make no sense.

After reading the data you have something less time than ~ 3ms to save the data.
Also the preallocate of a file is not possible with the long / variable duration.

The ~ 3ms could be enough to write over more than one block. Maybe you do not even need a continuous writing process. Did you test to acquire and save the data over a longer period of time?

In the other thread i have tried to read over blocks in less than 44 micros, so writing in 3000 micros can be possible.

By the way: how high is the resolution of your analog inputs?

Go Up