Pages: [1]   Go Down
Author Topic: [Solved] Yet another data logging problem...  (Read 2278 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Good evening.

I would like some guidance on logging data (writing it to a microSD card in the Ethernet Shield). I want to write data 100 times a second, so quite a lot faster than I could manage with a sketch I wrote today (attached, for reference). Each data point will be 14 bytes long, consisting of one long number (millis()), and 2 floats, separated by commas.

I have read several topics about writing to SD faster than 5 times a second. I read that the SDFat library was faster than the standard SD library, so I have downloaded the December version of SDFat. I am considering using a buffer after reading thread, but I don't know how I will write to buffer and then to the card. And I have read things about using 2 or even 3 buffers, and I am not sure if I will need that.

I will be using a Mega, so I should have enough RAM for a large array, but how big would it need to be? I read here that it was best to write in 512 bytes, so would this mean a buffer of exactly 512 bytes?

Thanks,
+-

* GraphDataLogger.ino (1.55 KB - downloaded 7 times.)
« Last Edit: April 17, 2012, 11:38:14 am by plusminus_ » Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 168
Posts: 12430
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Best performance is indeed made with buffers equal to the sector size of the SD card (512 bytes).

You say one record consist of 14 bytes, [millis, f1,f2] comma separated fields. But how do you separate records?

Consoder writing pure binary: (the commas do not add anything except 2/14 slack ~14%.
4 bytes long millis
4 bytes float f1
4 bytes float f2

So every 12 bytes is one record. 512 bytes can hold 512/12 = 42 records, leaving 8 bytes .

So in your main loop() you should have a counter that writes the 512 bytes buffer to SD after 42 records.
while writing this buffer you should fill buffer 2. etc

If you think reading 12 bytes records is difficult you could consider writing 16 bytes records of which exactly 32 fit in 512 bytes.
Room for an additional sensor smiley-wink
« Last Edit: April 08, 2012, 03:08:09 pm by robtillaart » Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1471
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

To log 1000 samples per second requires a totally different technique than your sketch.

I suggest you look at example sketches in fastLoggerBeta20110802.zip here http://code.google.com/p/beta-lib/downloads/list.

Forget advice like 512 byte buffers unless you are doing raw writes to the SD.

The big problem is that a write to a file on an SD can take 200 milliseconds.  You must capture data while the SD write is in progress.  This means interrupts, probably using timers, and cleaver buffering schemes.
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 168
Posts: 12430
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I thought that using 512 bytes buffers was allways a good idea, raw and cooked, as low level it will always be written raw ?

A question arose while thinking about your project:
If you do a 1000 samples per second, why add the millis() timestamp?
If evenly divided you just get consecutive numbers, if not you get multiple samples with the same timestamp (implying a zero duration between). using the micros as timestamp gives at least an - sub milli -  interval between two samples.




Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1471
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

For raw writes you must write 512 bytes.

For file writes there is some gain if all writes are 512 bytes and aligned on 512 byte boundaries.

For most apps this gain is not worth the effort.  My benchmark example in SdFat uses a 100 byte buffer. 

With a SanDisk 1 GB Ultra II card formatted FAT16 with the SdFormatter example I get this result:
Quote
Free RAM: 1029
Type is FAT16
File size 5MB
Starting write test.  Please wait up to a minute
Write 198.11 KB/sec
Maximum latency: 81396 usec, Avg Latency: 500 usec

Starting read test.  Please wait up to a minute
Read 299.33 KB/sec
Maximum latency: 2720 usec, Avg Latency: 329 usec

This is a very good card for data logging.  Not because of the average write rate of 198 KB/sec but the max latency of 81396 usec.  Some cards have a max write latency of 200000 usec.

The 198 KB/sec is way faster than needed to log 1000 records per second with 14 byte records.

Once again the design problem is to overcome the occasional long write latency that is inherent in SD cards.

Even class 10 cards have this problem.  The assumption is that devices like video cameras have lots of buffer so they achieve high average write rates for very large files.  This allows occasional pauses by the card to erase large areas of flash.  You can only write to erased flash.
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 168
Posts: 12430
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Quote
This allows occasional pauses by the card to erase large areas of flash.  You can only write to erased flash.

being a noob on the flash details: Can you erase flash in advance?
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1471
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You can use write commands that pre-erase but it only works for multiple block writes which can't be used for file writes.

SdFat can create large contiguous files which can be written this way.  See this post for more info http://arduino.cc/forum/index.php/topic,98898.msg744591.html#msg744591

My binaryLogger example in fastLoggerBeta20110802.zip http://code.google.com/p/beta-lib/downloads/list uses this method.  It can log 40,000 12 bit ADC samples per second.

I wrote an audio recorder that can record at 44,200 samples per second using the Arduino ADC in 8-bit mode.  It is located here http://code.google.com/p/waverp/.  It does raw write to pre-erased flash.

Logged

Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Best performance is indeed made with buffers equal to the sector size of the SD card (512 bytes).

You say one record consists of 14 bytes, [millis, f1,f2] comma separated fields. But how do you separate records?

Consider writing pure binary: (the commas do not add anything except 2/14 slack ~14%.
4 bytes long millis
4 bytes float f1
4 bytes float f2

So every 12 bytes is one record. 512 bytes can hold 512/12 = 42 records, leaving 8 bytes .

So in your main loop() you should have a counter that writes the 512 bytes buffer to SD after 42 records.
while writing this buffer you should fill buffer 2. etc

If you think reading 12 bytes records is difficult you could consider writing 16 bytes records of which exactly 32 fit in 512 bytes.
Room for an additional sensor smiley-wink
Hello, thanks for your reply!

I should explain what the data is for. My Arduino is serving a web page, and I'd like to have a live graph that plots the two values (the two floats) with time. I intend to use a JavaScript library called DyGraphs, which can use arrays or comma separated values to display multiple series, which is why the commas are in there for now.

So will each entry to the buffer contain the string with all 3 values at that time? Or just one long array? I tried to code it, but I didn't make much progress, I can't think straight right now...

P.S. This isn't strictly relevant, but... I was concerned that the script would not be able to read as the file is being written to, but I think with the right update rate that should not be a problem. I have a feeling I will need to overwrite existing data after a soft reset (e.g. when the page is reloaded) so that the graph doesn't contain old, irrelevant data, but I will sort that out later.

To log 1000 samples per second requires a totally different technique than your sketch.

I suggest you look at example sketches in fastLoggerBeta20110802.zip here http://code.google.com/p/beta-lib/downloads/list.

Forget advice like 512 byte buffers unless you are doing raw writes to the SD.

The big problem is that a write to a file on an SD can take 200 milliseconds.  You must capture data while the SD write is in progress.  This means interrupts, probably using timers, and cleaver buffering schemes.
Thanks for your reply!

Will I be able to use interrupts if I have other things going on in my sketch? Could I remove the code related to the ADC chip?

As I will be saving the data for a graph on a webpage, I can't save the data in binary format, and I will have to store it somewhere; I assumed a file on the SD card would be the 'best' place. I'm considering increasing my interval too, 1000 times a second for a graph seems unnecessary. I tried writing some code based on the SPI EEPROM example, but it went horribly wrong!

Code:
#include <SD.h>

char buffer[512]; // For storing data
int i = 0; // count
int r = 1234; // "reference"

int delta = 10; // sampling interval, ms
long previousMillis = 0;

const int chipSelect = 4;
const int yPin = A5; // motor output, after scaling

void setup() {
  Serial.begin(115200);
  Serial.print("Initialising SD card...");
  pinMode(10, OUTPUT); // CS pin
  pinMode(yPin, INPUT);

  // see if the card is present and can be initialised:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    return;
  }
  Serial.println("card initialised.");
}

void loop() {
  File dataFile = SD.open("graphlog.txt", FILE_WRITE);
  
  unsigned long currentMillis = millis();
  
  int y = analogRead(yPin);
  
  String graphString = String(millis());
  graphString += ",";
  graphString += String(r);
  graphString += ",";
  graphString += String(y);
  
  if(currentMillis - previousMillis == delta) {
    previousMillis = currentMillis;
    for (i=0; i < 510; i++) { // 512?
      buffer[i] = graphString;
    }
  }
}

void fill_buffer() {
  for (int i=0; i < 512; i++) {
    buffer[i] = i;
  }
}

Edit: I started typing the above when there were only 2 replies!

If you do a 1000 samples per second, why add the millis() timestamp?
If evenly divided you just get consecutive numbers, if not you get multiple samples with the same timestamp (implying a zero duration between). using the micros as timestamp gives at least an - sub milli -  interval between two samples.
I haven't found a way to specify the time between data points using DyGraphs, unfortunately, so I need to add the time so the script knows what to plot! I'm starting to think a much much lower sample rate would be more reasonable... this is all much more complicated than I anticipated!  smiley-razz

For raw writes you must write 512 bytes.

For file writes there is some gain if all writes are 512 bytes and aligned on 512 byte boundaries.

For most apps this gain is not worth the effort.  My benchmark example in SdFat uses a 100 byte buffer. 

With a SanDisk 1 GB Ultra II card formatted FAT16 with the SdFormatter example I get this result:

This is a very good card for data logging.  Not because of the average write rate of 198 KB/sec but the max latency of 81396 usec.  Some cards have a max write latency of 200000 usec.

The 198 KB/sec is way faster than needed to log 1000 records per second with 14 byte records.

Once again the design problem is to overcome the occasional long write latency that is inherent in SD cards.

Even class 10 cards have this problem.  The assumption is that devices like video cameras have lots of buffer so they achieve high average write rates for very large files.  This allows occasional pauses by the card to erase large areas of flash.  You can only write to erased flash.
I just ran bench with my little 256 MB card, formatted using SDFormatter. Here are the results:
Code:
Type any character to start
Free RAM: 1043
Type is FAT16
File size 5MB
Starting write test.  Please wait up to a minute
Write 165.30 KB/sec
Maximum latency: 107708 usec, Avg Latency: 600 usec

Starting read test.  Please wait up to a minute
Read 325.54 KB/sec
Maximum latency: 2336 usec, Avg Latency: 302 usec

By file write do you mean writing to a file (such as a CSV file)? If I don't need to have a 512 byte buffer, then would it just need to be long enough to store data points for more than the length of the maximum latency?
Logged

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1471
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Some examples I pointed out log in binary and some in csv.  Try the examples and study them.

Another way to log data at about 1000 samples per second is to use a RTOS.  I like ChibiOS and ported it to Arduino.  Look at the chMegaLogger example in ChibiOS20120221.zip here http://code.google.com/p/beta-lib/downloads/list.  It can log four analog pin every 1024 microseconds to a csv file.

Note that millis() ticks every 1024 microseconds except when it falls behind by a millisecond and then ticks twice to catch up so millis() is not too good as a clock for data logging.

You need to look at examples that are in the same ballpark as your requirement.  The Arduino SD.h data logging example that opens and closes a file in loop won't get you to 1% of your goal.  The Arduino group's examples are little league stuff when it comes to performance.  Their examples are great to get newbies started.
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 168
Posts: 12430
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

(back to the goal)
Can you explain why you need 1000 samples per second?
What is changing so fast you want to monitor?

maybe tell something more about the project?
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

(back to the goal)
Can you explain why you need 1000 samples per second?
What is changing so fast you want to monitor?

maybe tell something more about the project?
Good afternoon! Firstly I'd like to correct a mistake in the original post, I meant 100 times a second, not 1000! How embarrassing...

I am controlling a motor with an Arduino, which is serving a webpage where you can choose a target speed or angle, as well as gains for a PID controller (for example). I want to display a graph on the webpage which shows the reference value and the motor output against time. I planned to use a JavaScript graph library, which can read data from CSV or .txt files.

I plan to have my controller calculate a new control signal 100 times a second (though I may have to change this if I find that the Arduino can't keep up), and originally I wanted to save the reference value and the motor output at the same time. Now that I have thought about it, I don't think the graph needs to have data points so close together, so I could try saving data (millis, reference and output) to file 10 times a second. But the Arduino can't seem to write to file this quickly, so I thought a buffer and using SDFat would help. But I am still having trouble trying to use buffers.

Thanks and sorry for the mistake!
+-
Logged

Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote from: fat16lib
Hello!  smiley-grin

I sort of gave up on this for a few days, but then I went through fat16lib's examples and it looks like the AnalogLogger is pretty much exactly what I'm looking for! I will try to fuse it with my code tonight. Thank you fat16lib, your library and examples are just fantastic.

Could you tell me a little more about the ADC delay in the AnalogLogger sketch, please?
Logged

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1471
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

ADC_DELAY is for high impedance sensors.  If you have low impedance sources set ADC_DELAy to zero.

If ADC_DELAY is nonzero,  analogRead is called and the value is discarded to switch the ADC MUX to the channel then delay(ADC_DELAY) is called and finally analogRead() is called again to get the value.

This allows the allows the ADC sample and hold to stabilize with high impedance sensors.
Logged

Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

ADC_DELAY is for high impedance sensors.  If you have low impedance sources set ADC_DELAy to zero.

If ADC_DELAY is nonzero,  analogRead is called and the value is discarded to switch the ADC MUX to the channel then delay(ADC_DELAY) is called and finally analogRead() is called again to get the value.

This allows the allows the ADC sample and hold to stabilize with high impedance sensors.

Thanks for the explanation  smiley-grin
Logged

Offline Offline
Jr. Member
**
Karma: 1
Posts: 54
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Epilogue: A couple of nights ago while I was reading about HighCharts, I discovered that it could access data in other several formats, including JSON. I then discovered that I could use the TinyWebServer library to send data in this format, obviating (love that word) the need to write to SD rapidly and having the problem of erasing old irrelevant data.

So, I am using the JSON method instead; tests have shown that it works just as I want it to!  smiley-razz

But still, thank you for your help!
+-
Logged

Pages: [1]   Go Up
Jump to: