Pages: [1] 2   Go Down
Author Topic: Fast data logger for multiple analog pins  (Read 6965 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I have implemented a data logger to test several new features of SdFat and a small RTOS.  

This logger produces a CSV file and performs very well for a logger that formats text.  I have logged two analog pins at 1000 Hz on a Uno and 16 pins at 100 Hz on a Mega.

It is designed to log any number of analog pins starting with pin zero.  This could be modified easily.

It works best with a quality SD card but will provide good results with lower quality cards.

Several features of SdFat and the RTOS enhance performance.  

Two threads and a large buffer are used to overcome the occasional large write latency of SD cards.

A special version of analogRead() is used that sleeps during the ADC conversion.  This saves CPU time for the SD write thread.

A new SdFat function, printField(), is used to format data.  This function is several times faster than the standard Print class.

The logger is the nilSdLogger example in NilRTOS.

NilRTOS20130214.zip http://code.google.com/p/rtoslibs/downloads/list.

Three parameters configure the program.  Here is an example to log the 16 Mega analog pins at 100 Hz.
Code:
// Time between points in microseconds.
// Maximum value is 4,194,304 (2^22) usec.
const uint32_t PERIOD_USEC = 10000;

// Number of ADC channels to log
const uint8_t NADC = 16;

// FIFO buffer size. Adjust so unused idle thread stack is about 100 bytes.
const size_t FIFO_SIZE_BYTES = 4000;

Here is the beginning of the data file:
Quote
PERIOD_USEC,10000
ADC0,ADC1,ADC2,ADC3,ADC4,ADC5,ADC6,ADC7,ADC8,ADC9,ADC10,ADC11,ADC12,ADC13,ADC14,ADC15,Overruns
528,466,429,402,388,372,362,352,370,356,361,350,339,346,341,342,0
407,414,409,398,389,376,365,352,352,347,349,344,335,337,334,337,0
365,380,387,386,383,376,368,354,345,342,343,340,333,333,330,332,0
345,355,364,368,371,369,366,352,343,341,342,340,335,335,333,335,0
329,333,339,343,347,348,348,338,328,326,326,325,321,321,318,322,0
329,330,333,337,341,343,344,335,328,328,328,327,324,324,321,321,0
315,315,316,318,321,324,325,320,315,315,315,315,313,313,313,313,0
311,310,311,311,313,315,316,311,307,307,307,307,305,305,304,303,0
305,304,305,306,307,308,310,305,303,304,305,305,304,305,304,302,0
292,291,291,291,292,292,293,290,288,288,288,288,287,288,288,286,0

Statistics are printed at the end of a run.  Here are stats for the Mega.
Quote
Max Write Latency: 68780 usec
Unused Stack: 49 3207
FIFO record count: 125
Minimum free count: 117
A very good SD card was used so the max write latency was about 69 ms.  

The data read thread had 49 bytes of unused stack.

The SD write thread had 3207 unused bytes so more buffering could have been used.  Adjust FIFO_SIZE_BYTES so the write thread has about 100 unused bytes for best buffer performance.

125 buffers were allocated with space for 16 ADC value in each buffer.  The total buffer space was 4000  = 125*32 bytes.

The minimum free count was 117 buffers so there was never danger of a data overrun.  
« Last Edit: April 30, 2013, 08:46:52 am by fat16lib » Logged

Offline Offline
Newbie
*
Karma: 3
Posts: 39
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

can you tell the chances this would also work on an Arduino with a 328p ?
Thanks, Robert
Logged


0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yes it works on 328P Arduinos.  It has been used on 1284P in addition to Mega.

I have logged two or three analog pins at 1000 Hz on a Uno and all six pins at slower rates.  The maximum rate depends on your SD card.

All SD cards have occasional long write latencies.  Externally, SD cards appear to have simple 512 byte write blocks.  

Internally things are much more complex.  Physical blocks are much larger than 512 bytes and physical blocks are mapped to logical block addresses for flash wear leveling.  Flash must be erased before it can be written. This internal housekeeping can result in occasional long write times.

The SD spec allows write delays as long as 250 ms.  Most consumer cards have maximum write latencies around 100 ms.

You can run the SdFat bench example to get an idea of your card's write latency.  Here is an example from an 8 GB card.

Quote
Free RAM: 1027
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 192.05 KB/sec
Maximum latency: 88288 usec, Minimum Latency: 88 usec, Avg Latency: 515 usec
« Last Edit: April 30, 2013, 08:41:50 am by fat16lib » Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 29
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The example is great-- because it obviously works fast. I would like to experiment with it, but I have no idea how to connect an Arduino Due with an Sd card shield (I have a SEEED Studio SD card shield v 3.1). Could you please help me out with some details?

Kindest Regards,
MJM
Logged

0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I have no idea how to connect an Arduino Due
Sorry, I meant an Uno and have edited the original post.  It should work on your board.

It works on AVR Arduinos, 328, Mega, Leonardo, and 1284 boards.
« Last Edit: April 30, 2013, 02:55:24 pm by fat16lib » Logged

Global Moderator
Boston area, metrowest
Offline Offline
Brattain Member
*****
Karma: 538
Posts: 27069
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

2840? Did you intend 1284 there?
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
2840? Did you intend 1284 there?
Yes.  I will never get this post right.
Logged

Global Moderator
Boston area, metrowest
Offline Offline
Brattain Member
*****
Karma: 538
Posts: 27069
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

The post discussion is not that important, it's the logger code that matters smiley-wink
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

Austin, TEXAS
Offline Offline
Full Member
***
Karma: 0
Posts: 109
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi-

I like this development effort, good work!  I understand it too, multithreading and such is part of my day job, so no big deal there.  However I am still new to Arduino microcontrollers, so I am having a little difficulty understanding how to implement this in real ife.  I'd like to offer what I think you'd do and then have the experts tell me how wrong I am.  smiley-grin  If that's ok...

In my project I will want to datalog and I will be sampling an analog input (only one at the present time).  The 1000 samples/sec is more than I need, so this "fast datalogger" was/is what I was looking for to fill this need of the project.  So the plan included:

1) Getting the Adafruit datalogger shield and installing it on my Uno
2) Incorporating fat16lib's libraries and headers/sources into the build (presuming that is how they are used)
3) Utilizing this functionality as described in his documentation

Does that sound about right?  A few questions abound:

1) This uses I2C as I think I remember reading in the documentation.  I read somewhere else that "I2C is slow" ... if true, does that affect the overall ability of this datalogger hardware and software to keep pace with a 1000 samples/sec data rate?
2) I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support.  Is that correct?  It seems that someone posted that they were up-and-working with a 16G SD card and someone else was surprised that it was working.  If 2G isn't the number, is there such a limit?
3) It almost seems like (and please remember that I am new) the SD card in such a scenario is being used as a harddrive to store log files for later retrieval.  Is this customary in the Arduino microcontroller world?

I ask that last question (#3) because I have a very real need to log the data and then provide the ability for the user to retrieve it later on.  Seems easy enough, just remove the SD card, and pull it off on the PC, right?  However these components of mine will be encased in a plastic enclosure and I'd really prefer that the users didn't open it to remove the SD card to get their data.  So how could I allow them to access it?  Maybe by Wi-fi or Bluetooth?  Would there be any way to allow them to connect by a USB port and get their data, kinda like they would via a flashdrive/thumbstick/whatever-you-want-to-refer-to-it-as?

Many, many thanks!

Jerry
Logged

0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
This uses I2C as I think I remember reading in the documentation. 
No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.

Quote
I remember reading that 2G was about the max storage capability that these datalogger hardware devices were meant to support.
SdFat supports any SD card formatted FAT16 or FAT32.  SDHC cards can be as large as 32 GB.  The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32.  many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.
Logged

Austin, TEXAS
Offline Offline
Full Member
***
Karma: 0
Posts: 109
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

No, I2C is not used. The logger reads from the Arduino's internal ADC and write to the SD using the SPI bus.

Ok thanks, I couldn't remember if it was I2C or SPI.  Out of curiosity, you could use both I2C and SPI simultaneously (but to do different things) in your application couldn't you?  I wouldn't see why not, but I'm new, so...

SdFat supports any SD card formatted FAT16 or FAT32.  SDHC cards can be as large as 32 GB.  The maximum size of a file on a FAT32 volume is four GB so you must create multiple log files.

SdFat can write to SDXC cards that have been formatted FAT32.  many vendors sell 128 GB SDXC cards. FAT32 is not the standard format for SDXC cards but the SdFat formatting sketch, SdFormatter.ino, will format SDXC cards as FAT32 volumes and Windows, Macs, and Linux accept this format.

Good to know, thanks.  So how do folk who implement something like this typically retrieve their data?  Is it simply by removing the SD card and getting it directly from that?  It seems like I might need to get creative with this one.

Thanks again fat16lib for your work.
« Last Edit: May 02, 2013, 12:12:00 pm by av8or1 » Logged

Austin, TEXAS
Offline Offline
Full Member
***
Karma: 0
Posts: 109
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

fat16lib-

Allow me to ask another way.  Say you store your data to an SD card and you want to enable the user to get the data without removing the SD card from your device.  If you had a USB host breakout, how could you move the files from the SD card to the USB host such that a user could attach a flash drive/memory stick or their computer to the USB port and retrieve the files?

Thanks!
Logged

0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Why not locate the SD socket so the user can change the SD easily.  

Connecting a USB flash drive to an Arduino and transferring files to it really hard.  There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive.  If you could write the USB drive just log directly to it.  

I don't know of a clean way to move files from the SD to a PC/Mac over USB.

The ideal way would be for the SD on the Arduino to appear as an USB mass storage device.  This is also very difficult.

http://arduino.cc/forum/index.php?topic=143215.msg1075624

Paul Stoffregen has spent a lot of time on this.

The best I have done is send files over a USB serial connection or have the Arduino act as a web server using a SD/WiFi shield.

One SdFat user was trying to use an Eye-Fi card http://www.eye.fi/ I don't know if the user got it to work.
« Last Edit: May 06, 2013, 09:15:40 am by fat16lib » Logged

Austin, TEXAS
Offline Offline
Full Member
***
Karma: 0
Posts: 109
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Connecting a USB flash drive to an Arduino and transferring files to it really hard.  There is no point plugging a USB drive into the logger and transferring files from the SD to a flash drive.  If you could write the USB drive just log directly to it.  

Is there a way to log data to a USB flash drive with an Arduino?  I don't remember seeing one.  And your library uses the SD library to log data, so if there was a way to write to a USB drive, how would you log data to it?  That is, what function calls would you make?  Is there a library for all this stuff that I haven't found yet?

I'll look into the other options you mentioned, thanks.
Logged

0
Offline Offline
Edison Member
*
Karma: 64
Posts: 1634
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Is there a way to log data to a USB flash drive with an Arduino?

Not that I know about.  You would need a USB shield and a USB flash drive library.
Logged

Pages: [1] 2   Go Up
Jump to: