Fast data logger demo sketch

Using Arduino Storage
1

A number of people have asked how to log data at higher rates without dropping points. I have posted a demo sketch, fastLogger.pde, here Google Code Archive - Long-term storage for Google Code Project Hosting..

This sketch logs data from analog pin zero at over 2000 samples per second with a uniform time between samples.

It should work with almost any SD shield/module. Data is logged to a text file with one point per line.

This sketch requires the new beta of SdFat which is available at the above URL.

fastLogger.pde achieves high rates and is fairly easy to understand.

To log at higher rates you need to log binary data and use raw writes to pre-allocated contiguous files. This is far more complex than the method used by fastLogger.pde.

2

Thanks,

Rob

3

Wow, this is great!

I went away on vacation, planning to try & figure this out when I got back. And here it is.

So, TimerTwo you pass the (usec, function ptr, reset).

What does the reset boolean do?

I'll definitely be writing binary data. If one was to pre-allocate the file, how would you suggest keeping track of where the end of the file is? Can I keep another file open that simply writes how many blocks have been used every so often?

4

really good work, that you are doing there fat16lib.

as mentioned in this post

I'm using your contiguous logger as well and it works really fast!

do you think, that it's possible to interrupt the actual writing process? or would the chance of corrupting the data be too high?
I'm inches away from beeing actually Real Time capable with my logger - if i could interrupt the 900 µs write for 2 or 3 times, I'd be able to record all the CAN messages that are sent (up to 4 messages per millisecond)

plus, can I extend the preallocated file, or would i have to allocate a new one, when i reach the limit?

Have you tried both these things? if not, a simple opinion of yours, whether it could be possible or not would be greatly appreceated!

5

Concerning the interrupts -

i've done a little testing and it doesn't seem to be possible ... if i interrupt the writing it doesn't finish, even when i turn off the CS pin of the SD card ....

can you confirm this finding?

well ... there goes Real-Time capable :frowning:

edit:

i'm currently trying to selectively allow interrupts while writing to the SD card - but with no succes to this point ...

edit2:
it seeeems to work when I change the chipselect in the SD2Card.h code!!!!
until now, I haven't encountered an error and it seems to work!! ... whopee :smiley:

6

oscarcar,
Pre-allocating doesn't work well unless you do raw writes. When you rewrite a file SdFat must read the block, update it, and write it back. If you extend a file SdFat does not read the new blocks.

The reset argument controls the interval between interrupts. If reset is false, the interrupts occur evenly space every "usec" microseconds. If reset is true, the timer will be reset and the next interrupt will be "usec" after the user function "f()" returns.

keija.
I have not tried to interrupt SD multiple block writes. I would be interested how you changed chipselect.

7

Hooked it up tonight and it works flawlessly on an Arduino Mega with MEGA_SOFT_SPI set in the SdFatConfig.h header. I did bump it to 4000 samples/second, and I do get an occasional drop. I'll report back when I do some other optimizing. Although I didn't format yet, just deleted all files.

I don't think I explained myself well, or I am misunderstanding your answer.

I was thinking about pre-allocating a file with enough for 8 hours of data. But when transferring to a PC it would be nice to know where the true end of the file is so it could be truncated easily.
I was thinking a separate file that outputted occasionally how many blocks had been written. Not a big deal. I guess I could put a stop button to write "<\end>" or something.

8

fat16lib:
keija.
I have not tried to interrupt SD multiple block writes. I would be interested how you changed chipselect.

My µSD card reader quitted his job yesterday, so I can't confirm that it works yet - but as soon as I have evidence, I'll post my code!
What I can tell you already is, that I had to go into the SD2Card.cpp file and manually set chipselect low, allow interrupts and then turn them off again during the write process...

I would have never expected to go this deep into the materia ... wonders of arduino eh :wink:

9

oscarcar,

If you are using the Mega, you can make the ring buffer bigger. I tried 2000 entries during development with MEGA_SOFT_SPI set.

const uint16_t RING_DIM = 2000;

I was able to log 4000 samples per/sec without drops. I was formatting the samples as text. I didn't pre-allocate the file.

I have a few SD cards that do not work well in this application. They occasionally have a 200-300 ms write latency. Most SanDisk cards never have more than an occasional 120 ms latency. 2000 entries allows 500 ms at 4000 sample per second so even these cards work with the Mega.

Writing to a second file will kill performance. When I pre-allocate a file, I zero the file and then make sure each set of samples has at least one bit set. You can 'or' a bit into an ADC value that has less than 16-bits. Then you PC program can detect the end of samples. Just mask out this bit in the PC.

10

fat16lib,

excellent code! I tried to log for one hour and the binary file was quite compact and the acquisition of a 100 Hz sine wave quite precise.
Having the need to acquire three analog channels how would it be possible to modify your code to adapt it?
Any suggestion for a good approach?

thanks a lot!

11

Look at this program http://forum.arduino.cc/index.php?topic=228549.0.

12

Thanks a lot! Great code.
I tested it with a 1kHz sine wave on 5 channels and it works wonderfully.
It's also great the automatic adjustment of the prescaler.

Thanks again!!!!

13

Will this run with much higher sample rate on Arduino DUE?

14

Will this run with much higher sample rate on Arduino DUE?

FastLogger.ino only runs on AVR boards.

LowLatencyLogger https://github.com/greiman/SdFat/tree/master/SdFat/examples/LowLatencyLogger will run at about 4,000 data records per second on Due. A data record can have several sensor readings and is not limited to analog pins.

15

7 years late here .... but member fat16lib .... are you still around? Just wanted to quickly ask you if I'm doing the right thing in terms of getting all the libraries together to run your code on the Mega 2560.

I'm using a Mega 2560, and went to the google archive site you posted at the beginning of the thread. I first downloaded the 'fastLoggerBeta20110802.zip' file from Google Code Archive - Long-term storage for Google Code Project Hosting., and created a directory called 'fastlogger' under my 'arduino' directory, then put all those files in that zip file into my 'fastlogger' directory.

I also downloaded "SdFatBeta20130621.zip", and I extracted the files contained in the folder 'SdFat' (within that zip archive) and put all of the contents within SdFat into a folder of the same name (SdFat, under my 'libraries' folder).

I then loaded 'fastlogger.ino' into arduino sketch 1.8.3 ---- attempted to upload (compile etc) using arduino sketch, and I get these messages :

XXX\Documents\Arduino\fastlogger\binaryLogger.ino:27:22: fatal error: TimerTwo.h: No such file or directory
#include <TimerTwo.h>
compilation terminated.
exit status 1
Error compiling for board Arduino/Genuino Mega or Mega 2560.

I haven't changed any code that came with the archived files. Are there any extra details we need to follow in order to get this to load and work in the Mega 2560?

Is "TimerTwo" some kind of well known arduino library?
Also..... the "BufferedWriter" library ........ is that a well known arduino library?

And finally.... "McpSarAdc.h" ...... it's looking for this too. A search on google results in virtually nothing for this file.

Where do we get those from? I can't see them in that google archive area. I'm thinking that it would be nice if all of the libraries and include files were all properly bundled together (included and available in the same archive file) to reduce the amount of these compiling errors.

Thanks (if you're still around!)

[Also ..... one of your incredible pieces of code that ended up working on my mega 2560 is your AnalogBinLogger.ino sketch, from https://github.com/greiman/SdFat-beta/tree/master/SdFat/examples/AnalogBinLogger. For getting this to work, I needed to search on the net for your 'freestack.h' file in order to pop that into the directory along with the AnalogBinLogger.ino sketch. I also needed to change "csvStream.putc(',');" to this bracketted form "(csvStream.putc)(',');" in order to get the code to compile in Arduino sketch.]

16