Go Down

Topic: Try this super fast analog pin logger (Read 23 times) previous topic - next topic

fat16lib

Mar 25, 2014, 09:00 pm Last Edit: Mar 26, 2014, 12:30 pm by fat16lib Reason: 1
I have attached a beta version of a very fast data logger for analog pins.  I would appreciate any feedback from those who try this program.

The tests described below were done on an Uno.

Here is part of the readme file.
Quote

AnalogBinLogger.ino logs analog data to a binary SD file at high rates.

Samples are logged at regular intervals by using timer1.  Timer/Counter1
Compare Match B is used to trigger the ADC for the first pin in a sample.
The ADC is triggered for remaining sample pins in the ADC conversion complete
interrupt routine.

Data is captured in the ADC interrupt routine and saved in 512 byte buffers.

Buffered data is written to the SD in a function called from loop().  The
entire data set is written to a large contiguous file as a single multi-block
write.  This reduces write latency problems.

Many inexpensive SD cards work well at lower rates.  I used a $6.00
SanDisk 4 GB class 4 card for testing.

SanDisk class 4 cards work well at fairly high rates.  I used the 4 GB SanDisk
card to log a single pin at 40,000 samples per second.

The bintocsv folder contains a PC program for converting binary files to
CSV files.  I have included a executable for Windows.  Linux and Mac users
can build from the included source files.  bintocvs is a command line program.

bintocsv binFile csvFile


The attached file, DATA.png, is a plot of a 2 kHz sine wave logged at 40,000 samples per second.  FFT.png shows a FFT of this data.

I did a reliability test logging five analog pins at 5,000 samples per second.  This is an ADC rate of 25,000 values per second.  I logged 512 MB of data without dropping any values.

Serial output from the test.
Quote

Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG10.BIN
Max block write usec: 920
Record time sec: 10239.992
Sample count: 51199950
Samples/sec: 5000.00
Overruns: 0
Done


Here is the first part of the csv file produced from the test by the included bintocvs  program.
Quote

Interval,200.0000,usec
pin0,pin1,pin2,pin3,pin4
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0

Pins 0, 2, 4 are connected to ground, pin 1 to 5V and pin 3 to 3.3V.

Here is the configuration section of the logger.
Code: [Select]

//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers can be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 5000;  // Must be 0.25 or greater.

// The interval between samples in seconds, SAMPLE_INTERVAL, can be set to a
// constant instead of being calculated from SAMPLE_RATE.  SAMPLE_RATE is not
// used in the code below.  For example, setting SAMPLE_INTERVAL = 2.0e-4
// will result in a 200 microsecond sample interval.
const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;

// Setting ROUND_SAMPLE_INTERVAL non-zero will cause the sample interval to
// be rounded to a a multiple of the ADC clock period and will reduce sample
// time jitter.
#define ROUND_SAMPLE_INTERVAL 1
//------------------------------------------------------------------------------
// ADC clock rate.
// The ADC clock rate is normally calculated from the pin count and sample
// interval.  The calculation attempts to use the lowest possible ADC clock
// rate.
//
// You can select an ADC clock rate by defining the symbol ADC_PRESCALER to
// one of these values.  You must choose an appropriate ADC clock rate for
// your sample interval.
// #define ADC_PRESCALER 7 // F_CPU/128 125 kHz on an Uno
// #define ADC_PRESCALER 6 // F_CPU/64  250 kHz on an Uno
// #define ADC_PRESCALER 5 // F_CPU/32  500 kHz on an Uno
// #define ADC_PRESCALER 4 // F_CPU/16 1000 kHz on an Uno
// #define ADC_PRESCALER 3 // F_CPU/8  2000 kHz on an Uno (8-bit mode only)
//------------------------------------------------------------------------------
// Reference voltage.  See the processor data-sheet for reference details.
// uint8_t const ADC_REF = 0; // External Reference AREF pin.
uint8_t const ADC_REF = (1 << REFS0);  // Vcc Reference.
// uint8_t const ADC_REF = (1 << REFS1);  // Internal 1.1 (only 644 1284P Mega)
// uint8_t const ADC_REF = (1 << REFS1) | (1 << REFS0);  // Internal 1.1 or 2.56
//------------------------------------------------------------------------------
// File definitions.
//
// Maximum file size in blocks.
// The program creates a contiguous file with FILE_BLOCK_COUNT 512 byte blocks.
// This file is flash erased using special SD commands.  The file will be
// truncated if logging is stopped early.
const uint32_t FILE_BLOCK_COUNT = 256000;

// log file base name.  Must be six characters or less.
#define FILE_BASE_NAME "ANALOG"

// Set RECORD_EIGHT_BITS non-zero to only record the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 0
//------------------------------------------------------------------------------
// Pin definitions.
//
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;

// SD chip select pin.
const uint8_t SD_CS_PIN = SS;

Robert Getzner

Hi, looks great! On which Arduino / using which shield did you make your measurements ?
Thx, Robert
http://robertgetzner.com/PersonalWordpress/category/computer/arduino/

fat16lib

I tested on an Uno and a Mega.  The program is designed to work with other AVR boards.

To use other boards a section needs to add to this area to define BUFFER_BLOCK_COUNT and QUEUE_DIM.
Code: [Select]

#if defined(__AVR_ATmega328P__)
// 328 cpu -  use total of two 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 1;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 4;  // Must be a power of two!
//
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
// Mega - use total of 13 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 12;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 16;  // Must be a power of two!
//
#else  // CPU types
#error Undefined CPU
#endif  // CPU types


I plan to use the RAMEND symbol to define these symbols in the next version.

I tested with a number of SD shields/modules.  
You just need to edit this for your shield/module.
Code: [Select]

// SD chip select pin.
const uint8_t SD_CS_PIN = SS;


On Uno I mostly used an Adafruit data-logging shield.  It has two LEDs and I used one for an error indicator.
Code: [Select]

// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;

fat16lib

I have updated the zip file attached to the first post to AnalogBinLogger20140326beta.zip.

I now determine the amount of buffering by use of the RAMEND symbol.  This allows the logger to run on ATmega chips with 2 KB or more of SRAM.

I also fixed a problem that often causes an overrun due to SD write latency to be fatal.

I would appreciate any comments on results with various SD card.  I have had the best luck with SanDisk class 4 cards.

This is the card I used in development http://www.amazon.com/dp/B007JRB0RY.  I tested five of these cards, two 4 GB, two 8 GB, and one 32 GB.  I have had almost no overrun errors on an Uno with these cards.

I have done some testing with this microSD http://www.amazon.com/Sandisk-MicroSDHC-Memory-Card-Adapter/dp/B000WH6H1M It appears to work but I have no done a long reliability test.

Here is the result of a short 100 MB session with the 8 GB class 4 SanDisk  micro SD where the maximum write latency was 920 usec for a 512 byte block.
Quote

Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG01.BIN
Max block write usec: 920
Record time sec: 1993.845
Sample count: 9969226
Samples/sec: 5000.00
Overruns: 0


Here is one of the worst cards http://www.amazon.com/PNY-Optima-Class-Memory-P-SDHC4G4-EF/dp/B000L7INLU.  I use it when I want to be sure of overruns.

pito

Great results!  I wish I had the SdFat (or Fat16/12) running on the 8MB Ramdisk  :P

raschemmel

I tried the AnalogBinlogger program (installed the library) and it won't compile on an UNO.
I changed the "SS" to 10 for my SD card chip select for my SEEED STUDIO SD CARD shield.
I have a 1 Mb Sandisk formated with SDFAT16. Attached is the verbose compiler output pasted into WORD.
I read the Readme file and didn't really see anything that looked like configuration instructions.
Can anybody tell me what I overlooked ?
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

fat16lib

bintocsv is a PC program not an Arduino library.  Remove the bintocsv folder from  Arduino libraries.

raschemmel

Thanks !   That eliminated the compile errors. I moved the bintocsv folder to my Arduino reference data folder. I assume I need to remove the SD card from my reader, plug it into my PC, open a command line and then execute the command line command:
bintocsv binFile csvFile  ?  What's the easiest way to do it ? Should I copy the bin file(s) from the SD card to the bintocsv folder and execute the command as you entered it or do I need a path for anything? (windows 7)
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

fat16lib

I tend to put the bintocsv.exe in the folder with the .BIN file.

You could also use a path to another location where bintocsv.exe is located.

raschemmel

Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

raschemmel

Quote
Place the AnalogIsrLogger folder in your sketchbook folder. 


I found this instruction in the readme file but didn't see any folder with that spelling . Is this important ?

Also, regarding these instructions:
Quote
You must edit the configuration constants at the beginning of the program
to set the sample pins, sample rate, and other configuration values.


Which file do I edit with my C++ editor ?

Quote

Initially the program is setup to log the first five analog pins at 5000
samples per second.  Change these values to suit your needs.

See RateTable.txt for maximum allowed sample rates vs pin count and ADC clock
frequency.
The program has four commands:
c - convert file to CSV
d - dump data to Serial
e - overrun error details
r - record ADC data


I thought the bin to csv had to be performed using the command line instructions in the readme file .
So when you run the program it doesn't start recording (saving to SD) until you press "r" ?
The other two "d" & "e" make total sense.
Does the above mean I can perform any of those four commands after I start running the program ?
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

fat16lib

Quote

Quote
Place the AnalogIsrLogger folder in your sketchbook folder. 


I found this instruction in the readme file but didn't see any folder with that spelling . Is this important ?


I should have just said sketchbook I guess.  I didn't say the Arduino folder since you can select the folder to use.
Quote
The Arduino environment uses the concept of a sketchbook: a standard place to store your programs (or sketches). The sketches in your sketchbook can be opened from the File > Sketchbook menu or from the Open button on the toolbar. The first time you run the Arduino software, it will automatically create a directory for your sketchbook. You can view or change the location of the sketchbook location from with the Preferences dialog. 


Quote
Which file do I edit with my C++ editor ?

You edit AnalogBinLogger.ino.  for example to select the set of analog pins edit this area:
Code: [Select]

//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------


Quote
Does the above mean I can perform any of those four commands after I start running the program ?


Yes just type the letter corresponding to the command you want run.

Quote
I thought the bin to csv had to be performed using the command line instructions in the readme file .

It takes a long time to convert large binary files on the Arduino so the PC/Mac command line program is often required.

raschemmel

THANKS !
I found it .
Code: [Select]
//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers can be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 5000;  // Must be 0.25 or greater.

// The interval between samples in seconds, SAMPLE_INTERVAL, can be set to a
// constant instead of being calculated from SAMPLE_RATE.  SAMPLE_RATE is not
// used in the code below.  For example, setting SAMPLE_INTERVAL = 2.0e-4
// will result in a 200 microsecond sample interval.
const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;

// Setting ROUND_SAMPLE_INTERVAL non-zero will cause the sample interval to
// be rounded to a a multiple of the ADC clock period and will reduce sample
// time jitter.
#define ROUND_SAMPLE_INTERVAL 1
//------------------------------------------------------------------------------
// ADC clock rate.
// The ADC clock rate is normally calculated from the pin count and sample
// interval.  The calculation attempts to use the lowest possible ADC clock
// rate.
//
// You can select an ADC clock rate by defining the symbol ADC_PRESCALER to
// one of these values.  You must choose an appropriate ADC clock rate for
// your sample interval.
// #define ADC_PRESCALER 7 // F_CPU/128 125 kHz on an Uno
// #define ADC_PRESCALER 6 // F_CPU/64  250 kHz on an Uno
// #define ADC_PRESCALER 5 // F_CPU/32  500 kHz on an Uno
// #define ADC_PRESCALER 4 // F_CPU/16 1000 kHz on an Uno
// #define ADC_PRESCALER 3 // F_CPU/8  2000 kHz on an Uno (8-bit mode only)
//------------------------------------------------------------------------------
// Reference voltage.  See the processor data-sheet for reference details.
// uint8_t const ADC_REF = 0; // External Reference AREF pin.
uint8_t const ADC_REF = (1 << REFS0);  // Vcc Reference.
// uint8_t const ADC_REF = (1 << REFS1);  // Internal 1.1 (only 644 1284P Mega)
// uint8_t const ADC_REF = (1 << REFS1) | (1 << REFS0);  // Internal 1.1 or 2.56
//------------------------------------------------------------------------------
// File definitions.
//
// Maximum file size in blocks.
// The program creates a contiguous file with FILE_BLOCK_COUNT 512 byte blocks.
// This file is flash erased using special SD commands.  The file will be
// truncated if logging is stopped early.
const uint32_t FILE_BLOCK_COUNT = 256000;

// log file base name.  Must be six characters or less.
#define FILE_BASE_NAME "ANALOG"

// Set RECORD_EIGHT_BITS non-zero to only record the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 0
//------------------------------------------------------------------------------
// Pin definitions.
//
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;
 
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

raschemmel

I can't seem to get the program to run. It compiles fine but when I type in "r" in the serial port and hit enter, the error led
comes on and I get a "sample rate too fast" error
I attached the sketch renamed as MyAnalogBinLogger .
Can you help me configure it to sample A0 at 40,000 samples per second ?
I don't know what to select for ADC clock or SAMPLE_INTERVAL.
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

raschemmel

#14
Apr 13, 2014, 02:24 am Last Edit: Apr 13, 2014, 04:52 am by raschemmel Reason: 1
Got it to work by changing the ADC clock rate to 1000 hz . What does 8-bit mode mean for the 2000 hZ clock rate ?
see attached
Arduino UNOs, Pro-Minis, ATMega328, ATtiny85, LCDs, MCP4162, keypads,
DS18B20s,74c922,nRF24L01, RS232, SD card, RC fixed wing, quadcopter

Go Up