Go Down

Topic: Fast data logger for multiple analog pins (Read 8207 times) previous topic - next topic

fat16lib

Mar 11, 2013, 04:40 pm Last Edit: Apr 30, 2013, 03:46 pm by fat16lib Reason: 1
I have implemented a data logger to test several new features of SdFat and a small RTOS.  

This logger produces a CSV file and performs very well for a logger that formats text.  I have logged two analog pins at 1000 Hz on a Uno and 16 pins at 100 Hz on a Mega.

It is designed to log any number of analog pins starting with pin zero.  This could be modified easily.

It works best with a quality SD card but will provide good results with lower quality cards.

Several features of SdFat and the RTOS enhance performance.  

Two threads and a large buffer are used to overcome the occasional large write latency of SD cards.

A special version of analogRead() is used that sleeps during the ADC conversion.  This saves CPU time for the SD write thread.

A new SdFat function, printField(), is used to format data.  This function is several times faster than the standard Print class.

The logger is the nilSdLogger example in NilRTOS.

NilRTOS20130214.zip http://code.google.com/p/rtoslibs/downloads/list.

Three parameters configure the program.  Here is an example to log the 16 Mega analog pins at 100 Hz.
Code: [Select]

// Time between points in microseconds.
// Maximum value is 4,194,304 (2^22) usec.
const uint32_t PERIOD_USEC = 10000;

// Number of ADC channels to log
const uint8_t NADC = 16;

// FIFO buffer size. Adjust so unused idle thread stack is about 100 bytes.
const size_t FIFO_SIZE_BYTES = 4000;


Here is the beginning of the data file:
Quote

PERIOD_USEC,10000
ADC0,ADC1,ADC2,ADC3,ADC4,ADC5,ADC6,ADC7,ADC8,ADC9,ADC10,ADC11,ADC12,ADC13,ADC14,ADC15,Overruns
528,466,429,402,388,372,362,352,370,356,361,350,339,346,341,342,0
407,414,409,398,389,376,365,352,352,347,349,344,335,337,334,337,0
365,380,387,386,383,376,368,354,345,342,343,340,333,333,330,332,0
345,355,364,368,371,369,366,352,343,341,342,340,335,335,333,335,0
329,333,339,343,347,348,348,338,328,326,326,325,321,321,318,322,0
329,330,333,337,341,343,344,335,328,328,328,327,324,324,321,321,0
315,315,316,318,321,324,325,320,315,315,315,315,313,313,313,313,0
311,310,311,311,313,315,316,311,307,307,307,307,305,305,304,303,0
305,304,305,306,307,308,310,305,303,304,305,305,304,305,304,302,0
292,291,291,291,292,292,293,290,288,288,288,288,287,288,288,286,0


Statistics are printed at the end of a run.  Here are stats for the Mega.
Quote

Max Write Latency: 68780 usec
Unused Stack: 49 3207
FIFO record count: 125
Minimum free count: 117

A very good SD card was used so the max write latency was about 69 ms.  

The data read thread had 49 bytes of unused stack.

The SD write thread had 3207 unused bytes so more buffering could have been used.  Adjust FIFO_SIZE_BYTES so the write thread has about 100 unused bytes for best buffer performance.

125 buffers were allocated with space for 16 ADC value in each buffer.  The total buffer space was 4000  = 125*32 bytes.

The minimum free count was 117 buffers so there was never danger of a data overrun.  

can you tell the chances this would also work on an Arduino with a 328p ?
Thanks, Robert
http://robertgetzner.com/PersonalWordpress/category/computer/arduino/

fat16lib

#2
Mar 30, 2013, 04:31 pm Last Edit: Apr 30, 2013, 03:41 pm by fat16lib Reason: 1
Yes it works on 328P Arduinos.  It has been used on 1284P in addition to Mega.

I have logged two or three analog pins at 1000 Hz on a Uno and all six pins at slower rates.  The maximum rate depends on your SD card.

All SD cards have occasional long write latencies.  Externally, SD cards appear to have simple 512 byte write blocks.  

Internally things are much more complex.  Physical blocks are much larger than 512 bytes and physical blocks are mapped to logical block addresses for flash wear leveling.  Flash must be erased before it can be written. This internal housekeeping can result in occasional long write times.

The SD spec allows write delays as long as 250 ms.  Most consumer cards have maximum write latencies around 100 ms.

You can run the SdFat bench example to get an idea of your card's write latency.  Here is an example from an 8 GB card.

Quote

Free RAM: 1027
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 192.05 KB/sec
Maximum latency: 88288 usec, Minimum Latency: 88 usec, Avg Latency: 515 usec

mjmes

The example is great-- because it obviously works fast. I would like to experiment with it, but I have no idea how to connect an Arduino Due with an Sd card shield (I have a SEEED Studio SD card shield v 3.1). Could you please help me out with some details?

Kindest Regards,
MJM

fat16lib

#4
Apr 30, 2013, 03:46 pm Last Edit: Apr 30, 2013, 09:55 pm by fat16lib Reason: 1
Quote
I have no idea how to connect an Arduino Due

Sorry, I meant an Uno and have edited the original post.  It should work on your board.

It works on AVR Arduinos, 328, Mega, Leonardo, and 1284 boards.

CrossRoads

Designing & building electrical circuits for over 25 years.  Screw Shield for Mega/Due/Uno,  Bobuino with ATMega1284P, & other '328P & '1284P creations & offerings at  my website.

fat16lib

Quote

2840? Did you intend 1284 there?

Yes.  I will never get this post right.

CrossRoads

The post discussion is not that important, it's the logger code that matters ;)
Designing & building electrical circuits for over 25 years.  Screw Shield for Mega/Due/Uno,  Bobuino with ATMega1284P, & other '328P & '1284P creations & offerings at  my website.

av8or1

Hi-

I like this development effort, good work!  I understand it too, multithreading and such is part of my day job, so no big deal there.  However I am still new to Arduino microcontrollers, so I am having a little difficulty understanding how to implement this in real ife.  I'd like to offer what I think you'd do and then have the experts tell me how wrong I am.  :D  If that's ok...

In my project I will want to datalog and I will be sampling an analog input (only one at the present time).  The 1000 samples/sec is more than I need, so this "fast datalogger" was/is what I was looking for to fill this need of the project.  So the plan included:

1) Getting the Adafruit datalogger shield and installing it on my Uno
2) Incorporating fat16lib's libraries and headers/sources into the build (presuming that is how they are used)
3) Utilizing this functionality as described in his documentation

Does that sound about right?  A few questions abound:

1) This uses I2C as I think I remember reading in the documentation.  I read somewhere else that "I2C is slow" ... if true, does that affect the overall ability of this datalogger hardware and software to keep pace with a 1000 samples/sec data rate?
2) I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support.  Is that correct?  It seems that someone posted that they were up-and-working with a 16G SD card and someone else was surprised that it was working.  If 2G isn't the number, is there such a limit?
3) It almost seems like (and please remember that I am new) the SD card in such a scenario is being used as a harddrive to store log files for later retrieval.  Is this customary in the Arduino microcontroller world?

I ask that last question (#3) because I have a very real need to log the data and then provide the ability for the user to retrieve it later on.  Seems easy enough, just remove the SD card, and pull it off on the PC, right?  However these components of mine will be encased in a plastic enclosure and I'd really prefer that the users didn't open it to remove the SD card to get their data.  So how could I allow them to access it?  Maybe by Wi-fi or Bluetooth?  Would there be any way to allow them to connect by a USB port and get their data, kinda like they would via a flashdrive/thumbstick/whatever-you-want-to-refer-to-it-as?

Many, many thanks!

Jerry

fat16lib

Quote

This uses I2C as I think I remember reading in the documentation. 

No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.

Quote

I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support.

SdFat supports any SD card formatted FAT16 or FAT32.  SDHC cards can be as large as 32 GB.  The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32.  many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.

av8or1

#10
May 02, 2013, 07:10 pm Last Edit: May 02, 2013, 07:12 pm by av8or1 Reason: 1

No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.


Ok thanks, I couldn't remember if it was I2C or SPI.  Out of curiosity, you could use both I2C and SPI simultaneously (but to do different things) in your application couldn't you?  I wouldn't see why not, but I'm new, so...


SdFat supports any SD card formatted FAT16 or FAT32.  SDHC cards can be as large as 32 GB.  The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32.  many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.


Good to know, thanks.  So how do folk who implement something like this typically retrieve their data?  Is it simply by removing the SD card and getting it directly from that?  It seems like I might need to get creative with this one.

Thanks again fat16lib for your work.

av8or1

fat16lib-

Allow me to ask another way.  Say you store your data to an SD card and you want to enable the user to get the data without removing the SD card from your device.  If you had a USB host breakout, how could you move the files from the SD card to the USB host such that a user could attach a flash drive/memory stick or their computer to the USB port and retrieve the files?

Thanks!

fat16lib

#12
May 06, 2013, 03:59 pm Last Edit: May 06, 2013, 04:15 pm by fat16lib Reason: 1
Why not locate the SD socket so the user can change the SD easily.  

Connecting a USB flash drive to an Arduino and transferring files to it really hard.  There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive.  If you could write the USB drive just log directly to it.  

I don't know of a clean way to move files from the SD to a PC/Mac over USB.

The ideal way would be for the SD on the Arduino to appear as an USB mass storage device.  This is also very difficult.

http://arduino.cc/forum/index.php?topic=143215.msg1075624

Paul Stoffregen has spent a lot of time on this.

The best I have done is send files over a USB serial connection or have the Arduino act as a web server using a SD/WiFi shield.

One SdFat user was trying to use an Eye-Fi card http://www.eye.fi/ I don't know if the user got it to work.

av8or1


Connecting a USB flash drive to an Arduino and transferring files to it really hard.  There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive.  If you could write the USB drive just log directly to it.  


Is there a way to log data to a USB flash drive with an Arduino?  I don't remember seeing one.  And your library uses the SD library to log data, so if there was a way to write to a USB drive, how would you log data to it?  That is, what function calls would you make?  Is there a library for all this stuff that I haven't found yet?

I'll look into the other options you mentioned, thanks.

fat16lib

Quote
Is there a way to log data to a USB flash drive with an Arduino?


Not that I know about.  You would need a USB shield and a USB flash drive library.

Go Up