Aquisition rate with SD Card

Hello!

In a project I’m designing, I need to make aquisition of three sensors with a frequency of 333.3333… Hz (delays of 3 ms in 1 second), display it in the serial monitor and save it to and SD card.

My question is if saving data to SD card(it would be one println per each aquisition of the three sensors) makes the aquisition rate being changed? It is really important to me to have a minimum of 300Hz!

Thanks :slight_smile:

The code I wrote is the following:

//aquisition variables
const int EMG_pin = A0; // Analog input pin that the EMG signal is attached to
long aq_delay = 3; // Signal aquisition rate;
unsigned long last_aq_time = 0;
int EMG_value;

//SD Card libraries & variables
#include <SPI.h>
#include <SD.h>
File myFile;

void setup() {
// put your setup code here, to run once:
Serial.begin(9600);

while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}

Serial.print(“Initializing SD card…”);

if (!SD.begin(4)) {
Serial.println(“initialization failed!”);
return;
}
Serial.println(“initialization done.”);

// open the file. note that only one file can be open at a time,
// so you have to close this one before opening another.
myFile = SD.open(“aquisition.txt”, FILE_WRITE);

}

void loop() {
// put your main code here, to run repeatedly:
if(millis() - last_aq_time >= aq_delay){ //establishes the aquisition rate
last_aq_time = millis();
//signal aquisition
EMG_value = analogRead(EMG_pin);

//data display
Serial.println(EMG_value);
//data saving
if (myFile) {
myFile.println(EMG_value);
// close the file:
myFile.close();
} else {
// if the file didn’t open, print an error:
Serial.println(“error opening test.txt”);
}

}
}

diogotec: make aquisition of three sensors with a frequency of 333.3333... Hz (delays of 3 ms in 1 second), display it in the serial monitor ...........

I think you should re-think that, it might help with your SD recording problem.

What do I need to rethink exactly? The aqusition frequency has to be higher than 300Hz (according to Nyquist theorem).

Maybe you can read my post in another thread.

I think the direct writing on sd card's without buffering is awkward.

I can only report from the reading, but the writing will be similar / even more difficult. If you write over several blocks (512 byte), you usually have long waiting times (~ 2-3 ms). In another post, I have created a continuous reading process. This can read a 512 byte block in 1.05ms. However always a complete 512 byte block must be read.

If you want to write the data to a sd card and this over a longer period (and a buffer in the arduino ram would not be enough), I would definitely put on a continuous writing process! Possibly, it is enough to preallocate the target file.

diogotec:
What do I need to rethink exactly?
The aqusition frequency has to be higher than 300Hz (according to Nyquist theorem).

You need to rethink the wisdom of sending data to a serial monitor @ 300 Hz. I don’t know who the hell Nyquist is, but I bet he can’t read a serial monitor updating data at 300Hz. I’m also betting you can’t either. Nobody would have the faintest idea of what you are trying to do, or why, but it would appear that your whole design approach is suss. If you re-thought the sanity of the data to the monitor, you might get Arduino to turn out something useful thereto, and you might then have an easier time feeding that same data to the SD.

@Nick_Pyner, eletromyography signal (signal from muscles) requires such aquisition rate..............

@DrDoom thanks for your reply! Sorry, but I still have questions.

1) About the buffer you are right! But I didn't understanded the 512 byte part (you said it takes longer but you recomend to do it?). How can I do that? I want to write the time + 3 sensors in one line like this: HOUR 1022 242 4234 Is it possible with what you are saying?

It's not for me to dispute that, I have never heard of eletromyography. I'm talking about the wisdom of sending the data to a monitor 300 times a second, which is very likely just a dog-chasing-car situation, even if it is possible. Feel free to explain to me how good an idea it is because, right now, I think it is a seriously stupid one. Indeed, I would go as far as to say that at this stage, the only redeeming feature of the project being designed that is evident, is the USB cable connecting Arduino to serial monitor. This cable offers the opportunity to connect Arduino to a proper terminal, and thereby record the data direct to the PC hard drive, hence making the SD card redundant. Maybe DrDoom has some pertinent comment but, along with my bets about your ability to read the serial monitor, I'm also willing to bet the aquisition rate of a PC hard drive will be somewhat better than that of an SD card. This does not necessarily solve your problem, it may be simply moving it, and you may have to get your microprocessor to do some microprocessing - before you send it anywhere..

Nick_Pyner: It's not for me to dispute that, I have never heard of eletromyography. I'm talking about the wisdom of sending the data to a monitor 300 times a second, which is very likely just a dog-chasing-car situation, even if it is possible. Feel free to explain to me how good an idea it is because, right now, I think it is a seriously stupid one. Indeed, I would go as far as to say that at this stage, the only redeeming feature of the project being designed that is evident, is the USB cable connecting Arduino to serial monitor. This cable offers the opportunity to connect Arduino to a proper terminal, and thereby record the data direct to the PC hard drive, hence making the SD card redundant. Maybe DrDoom has some pertinent comment but, along with my bets about your ability to read the serial monitor, I'm also willing to bet the aquisition rate of a PC hard drive will be somewhat better than that of an SD card. This does not necessarily solve your problem, it may be simply moving it, and you may have to get your microprocessor to do some microprocessing - before you send it anywhere..

Signals that come from the body usually have high frequencies. For example, for recording the heart beat u need at least to record it at 100Hz. The reason I want to record my data to an SD card is because I want to make my system portable.

diogotec: Signals that come from the body usually have high frequencies.

I'm afraid that this makes not the slightest difference to what I said.

display it in the serial monitor and save it to and SD card.

and

The reason I want to record my data to an SD card is because I want to make my system portable.

Good luck with that. And somehow I thought you wanted to display it on the serial monitor. Don't forget to disconnect it.

I have also seen this post: http://forum.arduino.cc/index.php?topic=49649.msg354701#msg354701 But I don't know how to apply it to my case, and it seems a bit old, I don't know if there are new methods.

Used the following code to test how much time it takes 19 ms to aquire data from the 3 sensores, print it to the serial and to the SD card (see code!) using O_CREAT | O_WRITE.

void setup() { // put your setup code here, to run once: Serial.begin(9600);

while (!Serial) { ; // wait for serial port to connect. Needed for native USB port only }

Serial.print("Initializing SD card...");

if (!SD.begin(4)) { Serial.println("initialization failed!"); return; } Serial.println("initialization done.");

// open the file. note that only one file can be open at a time, // so you have to close this one before opening another. myFile = SD.open("asd.txt", O_CREAT | O_WRITE);

}

void loop() { // put your main code here, to run repeatedly: if(millis() - last_aq_time >= aq_delay){ //establishes the aquisition rate last_aq_time = millis(); //signal aquisition line = analogRead(EMG_pin) + tabe + analogRead(EMG_pin) + tabe + analogRead(EMG_pin);

//data display Serial.println(line); Serial.println(millis()); //data saving if (myFile) { myFile.println(line); Serial.println(millis()); } else { // if the file didn't open, print an error: Serial.println("error opening test.txt"); }

}

} [\QUOTE]

Also, I don't know how I can store the three sensor values in a buffer! I know nothing about buffers.. Please help :x

Ok, first let us know more about your requirements.

So far we know that you want to store 3 sensor values ​​with a frequency of at least 300 Hz. Show + save. Even if ur first code only shows 1 sensor pin/value (const int EMG_pin = A0;).

OK. But how long will the values ​​be processed? 1 second? 10 seconds? 1 minute? 10 minutes or 1 hour? Is the measurement limited to a fixed duration or is the duration variable? Do you want to use your project professionally or is it a demo for the school / university? That explains how much time and work you want to invest. All this can lead to a possibly different implementation.

A buffer is nothing but a memory that temporarily stores data before it is processed or stored. So if you get your sensor values, they can either be displayed / saved directly, or you can only save it first. 100 values ​​before you display / save them.
Like this e.g.:

const int sensorBufferSize = 100;
uint16_t  sensorBuffer[sensorBufferSize][3];//2 Dimensional Array -> Buffer with 100x3 Values
int       sensorBufferIndex = 0;//where to store next sensor values in the buffer?
...
      if(sensorBufferIndex < sensorBufferSize){
        sensorBuffer[sensorBufferIndex][0] = analogRead(PIN_EMG_1);
        sensorBuffer[sensorBufferIndex][1] = analogRead(PIN_EMG_2);
        sensorBuffer[sensorBufferIndex][2] = analogRead(PIN_EMG_3);
        sensorBufferIndex++;
      } else {
        //TODO: Warning, Buffer Overflow!!! the buffer with 100 different values is filled up, cant store more, need to write to sd card for example!
      }

So u can hold some values bevor writing them. I think u should read this thread from this post. I have checked the performance for reading an sd card. writing is similar.

But I didn’t understanded the 512 byte part

In the thread / post i have linked, i tested to read less and more than 512 byte.

The results explain the behavior.
A sd card contains many small 512 byte blocks. A file is distributed to many of these blocks. The loading / creation of a block takes time.
In order to avoid the problem as far as possible, one can use continuous reading and writing. The time for reading / creating blocks is thereby greatly minimized. After all, you have maximum <3ms time to write the data.
I guess you can write the first 512 byte without problems. After the first 512 byte, you will have to wait a while for the writing process.

Thanks for your reply and help DrDooom!

But how long will the values ​​be processed? 1 second? 10 seconds? 1 minute? 10 minutes or 1 hour? Is the measurement limited to a fixed duration or is the duration variable? Do you want to use your project professionally or is it a demo for the school / university? That explains how much time and work you want to invest. All this can lead to a possibly different implementation.

The objective of my project is to create a professional portable device (this explains why I want to save it to SD card), that can be used 24h+ without stopping (duration variable!).

So u can hold some values bevor writing them. I think u should read this thread from this post. I have checked the performance for reading an sd card. writing is similar.

So, according to your tests, using the SDfat Bibliothek with the File Class and a buffer improves speed, right?

The code you show is a buffer containing 100x3 elements, but how can I save only 512 bytes?

Well, that was just a simple example to buffer data. Data buffering can ensure that writing to a sd card does not delay your 300 Hz interval. If you want to save data permanently and over a long period, the buffer will make no sense.

After reading the data you have something less time than ~ 3ms to save the data. Also the preallocate of a file is not possible with the long / variable duration.

The ~ 3ms could be enough to write over more than one block. Maybe you do not even need a continuous writing process. Did you test to acquire and save the data over a longer period of time?

In the other thread i have tried to read over blocks in less than 44 micros, so writing in 3000 micros can be possible.

By the way: how high is the resolution of your analog inputs?

If you want to save data permanently and over a long period, the buffer will make no sense.

Why not?

The ~ 3ms could be enough to write over more than one block. Maybe you do not even need a continuous writing process. Did you test the data over a longer period of time?

I'm afraid I did not understand what you meant!

By the way: how high is the resolution of your analog inputs?

I'm using arduino UNO, so 10 bits resolution (0-1023).

The sensors are eletromyography and a piezoeletric sensor (both I don't expect to reach 4 digits number), and an accelerometer. With the accelerometer I only want to save if the person is up (1) or layed down (0).

If you want to save data permanently and over a long period, the buffer will make no sense.

Why not?

Well, with a buffer u are collecting data and than u write once every 500 ms. but u have to write e.g. 1000 bytes. this can take longer than ur 3ms (300 Hz). the writing to sd card clearly blocks ur arduino (the loop waits for the sd card writing) and you can't continue with the next sensor aequirement.

so it is better to write after every sensor aequiring. than u need to write e.g. 10 bytes and this should be possible in 3ms. so the arduino will resume with the next loop / and aequire after 3 ms.

What you are basically saying is that to use the writing to the SDcard (print + flush?) in order to make the 3ms delay? Using the SDfat library and the SdFile Class? Can you help me how to use this library?

I have written a test tool to acquire 3 analog values ​​and check the performance for reading / saving data (test result at the end). Hmmm, I do not know why, but the writing performance looks like there are no big latencies like reading over blocks. Maybe I was wrong when I said that there are problems with writing to several blocks.

Before you deal with the writing, however, you should consider how you most effectively save your sensor data.

I think you would most likely save the integer values ​​ascii text?

800,920,125
150,193,740
102,591,690
80,1015,1002

But you would have to save for the 3 values ​​(0-1023) + separators up to 15 bytes. 1023,1023,1023\n (total 15 chars)

It is more effective to save the data as bytes. Since you have 10 bit values ​​(0-1023), you can not save them as a byte (8 bit, 0-255). You need 2 byte's. There is a good possibility to convert a 16 bit variable into 2x 8 bit variable. I recommend you read something about bit shifting. So if you can store a value (10 bit) in two byte, you need for 3 sensor values ​​only total 6 byte. The speed and the overall data size is considerably reduced.

Save as Text: 15 byte * 333 Hz * 60 sec * 60 min * 24 Hours = ~ 412 MB Save as Bytes: 6 byte * 333 Hz * 60 sec * 60 min * 24 Hours = ~ 165 MB

Here is the output of the test (data saved as 6 byte array):

Setup

  • Serial Buffer: 63 bytes
  • Pin Modes
  • Set Pins High/Low
  • Init SD Card done ### Setup finished ### Starting acquire...
  • open sensor data file / truncate done! size 0 byte Stopped acquire!

duration test: 1148 sec (19.13 minutes) debugCount: 382915x sensor data acquired debugSensorMin: 1270 micros (1.27 ms) -> min duration to acquire sensor data debugSensorMax: 1274 micros (1.27 ms) -> max duration to acquire sensor data debugSensorTotal: 486552494 micros (486552.49 ms) -> total duration to acquire sensor data debugFileWriteMin: 8 micros (0.01 ms) -> min duration to write the 6 byte data to sd card debugFileWriteMax: 9 micros (0.01 ms) -> max duration to write the 6 byte data to sd card debugFileWriteTotal: 8151947 micros (8151.95 ms) -> total duration to write the 6 byte data to sd card debugFileWriteByteCount: 2297490 bytes (2243.64 kb) -> total bytes written zu sd card avg sensor read: 1270.65 micros (1.27 ms) avg file write: 21.29 micros (0.02 ms) perf. file write: 275.23 kb/sec

Total time 1148 sec and 382915 sensor acquire: the frequency is ~333,54 Hz over 19 Minutes.

The file size on sd card with raw byte data:

 Datenträger in Laufwerk F: ist 2GB SD CARD
 Volumeseriennummer: CC86-F45B

01.01.2000  01:00         2.297.490 sensor.dat
               1 Datei(en),      2.297.490 Bytes
               0 Verzeichnis(se),  1.979.711.488 Bytes frei

So saving without a buffer should be no problem!

WOW! DrDooom, I am very grateful for you detailed help! Although I still have some pending questions!

1) I have read and learned about bit-shifting. I used the following code to obtain the 2 bytes out of my sensor value:

byte EMG_byte[2]; //array containing the EMG sensor reading
EMG_byte[0] = (byte) ((EMG_value >> 8) & 0xFF);
EMG_byte[1] = (byte) (EMG_value & 0xFF);

I have tested it using Serial.print, which outputs the bytes in decimal, which I then convert to binary, merge, and convert to decimal, and it works.

Tho, my problem is that I don't know how to print these values to the SD.card. I have tried the following:

myFile.write(EMG_byte,2);

but I get the following in the txt file: http://imgur.com/HCN48Yn Can you help me out?

2) In your report you have:

Here is the output of the test (data saved as 6 byte array):

Setup

  • Serial Buffer: 63 bytes
  • Pin Modes
  • Set Pins High/Low
  • Init SD Card done ### Setup finished ### Starting acquire...
  • open sensor data file / truncate done! size 0 byte Stopped acquire!

In the setup why do you talk about a buffer of 63 bytes? Also, in the acquiring section, I am supposed to: 1 - open the file ONCE 2 - write data to it (and flush???) 3 - at the end, 24hours later or so, close the file?

3) In your simulation you used what library?

Thank you very much!!