Arduino Forum

Using Arduino => Storage => Topic started by: fat16lib on Mar 25, 2014, 09:00 pm

Title: Try this super fast analog pin logger
Post by: fat16lib on Mar 25, 2014, 09:00 pm
I have removed the beta version of this program.  Please download the newest SdFat from GitHub https://github.com/greiman/SdFat (https://github.com/greiman/SdFat).

AnalogBinLogger is now a SdFat example https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger (https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger).

Also see this directory https://github.com/greiman/SdFat/tree/master/AnalogBinLoggerExtras (https://github.com/greiman/SdFat/tree/master/AnalogBinLoggerExtras)

The tests described below were done on an Uno.

Here is part of the readme file.
Quote
AnalogBinLogger.ino logs analog data to a binary SD file at high rates.

Samples are logged at regular intervals by using timer1.  Timer/Counter1
Compare Match B is used to trigger the ADC for the first pin in a sample.
The ADC is triggered for remaining sample pins in the ADC conversion complete
interrupt routine.

Data is captured in the ADC interrupt routine and saved in 512 byte buffers.

Buffered data is written to the SD in a function called from loop().  The
entire data set is written to a large contiguous file as a single multi-block
write.  This reduces write latency problems.

Many inexpensive SD cards work well at lower rates.  I used a $6.00
SanDisk 4 GB class 4 card for testing.

SanDisk class 4 cards work well at fairly high rates.  I used the 4 GB SanDisk
card to log a single pin at 40,000 samples per second.

The bintocsv folder contains a PC program for converting binary files to
CSV files.  I have included a executable for Windows.  Linux and Mac users
can build from the included source files.  bintocvs is a command line program.

bintocsv binFile csvFile
The attached file, DATA.png, is a plot of a 2 kHz sine wave logged at 40,000 samples per second.  FFT.png shows a FFT of this data.

I did a reliability test logging five analog pins at 5,000 samples per second.  This is an ADC rate of 25,000 values per second.  I logged 512 MB of data without dropping any values.

Serial output from the test.
Quote
Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG10.BIN
Max block write usec: 920
Record time sec: 10239.992
Sample count: 51199950
Samples/sec: 5000.00
Overruns: 0
Done
Here is the first part of the csv file produced from the test by the included bintocvs  program.
Quote
Interval,200.0000,usec
pin0,pin1,pin2,pin3,pin4
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
Pins 0, 2, 4 are connected to ground, pin 1 to 5V and pin 3 to 3.3V.

Here is the configuration section of the logger.
Code: [Select]

//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers can be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 5000;  // Must be 0.25 or greater.

// The interval between samples in seconds, SAMPLE_INTERVAL, can be set to a
// constant instead of being calculated from SAMPLE_RATE.  SAMPLE_RATE is not
// used in the code below.  For example, setting SAMPLE_INTERVAL = 2.0e-4
// will result in a 200 microsecond sample interval.
const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;

// Setting ROUND_SAMPLE_INTERVAL non-zero will cause the sample interval to
// be rounded to a a multiple of the ADC clock period and will reduce sample
// time jitter.
#define ROUND_SAMPLE_INTERVAL 1
//------------------------------------------------------------------------------
// ADC clock rate.
// The ADC clock rate is normally calculated from the pin count and sample
// interval.  The calculation attempts to use the lowest possible ADC clock
// rate.
//
// You can select an ADC clock rate by defining the symbol ADC_PRESCALER to
// one of these values.  You must choose an appropriate ADC clock rate for
// your sample interval.
// #define ADC_PRESCALER 7 // F_CPU/128 125 kHz on an Uno
// #define ADC_PRESCALER 6 // F_CPU/64  250 kHz on an Uno
// #define ADC_PRESCALER 5 // F_CPU/32  500 kHz on an Uno
// #define ADC_PRESCALER 4 // F_CPU/16 1000 kHz on an Uno
// #define ADC_PRESCALER 3 // F_CPU/8  2000 kHz on an Uno (8-bit mode only)
//------------------------------------------------------------------------------
// Reference voltage.  See the processor data-sheet for reference details.
// uint8_t const ADC_REF = 0; // External Reference AREF pin.
uint8_t const ADC_REF = (1 << REFS0);  // Vcc Reference.
// uint8_t const ADC_REF = (1 << REFS1);  // Internal 1.1 (only 644 1284P Mega)
// uint8_t const ADC_REF = (1 << REFS1) | (1 << REFS0);  // Internal 1.1 or 2.56
//------------------------------------------------------------------------------
// File definitions.
//
// Maximum file size in blocks.
// The program creates a contiguous file with FILE_BLOCK_COUNT 512 byte blocks.
// This file is flash erased using special SD commands.  The file will be
// truncated if logging is stopped early.
const uint32_t FILE_BLOCK_COUNT = 256000;

// log file base name.  Must be six characters or less.
#define FILE_BASE_NAME "ANALOG"

// Set RECORD_EIGHT_BITS non-zero to only record the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 0
//------------------------------------------------------------------------------
// Pin definitions.
//
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;

// SD chip select pin.
const uint8_t SD_CS_PIN = SS;
Title: Re: Try this super fast analog pin logger
Post by: RobertGetzner on Mar 25, 2014, 09:42 pm
Hi, looks great! On which Arduino / using which shield did you make your measurements ?
Thx, Robert
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 25, 2014, 10:06 pm
I tested on an Uno and a Mega.  The program is designed to work with other AVR boards.

To use other boards a section needs to add to this area to define BUFFER_BLOCK_COUNT and QUEUE_DIM.
Code: [Select]

#if defined(__AVR_ATmega328P__)
// 328 cpu -  use total of two 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 1;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 4;  // Must be a power of two!
//
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
// Mega - use total of 13 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 12;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 16;  // Must be a power of two!
//
#else  // CPU types
#error Undefined CPU
#endif  // CPU types


I plan to use the RAMEND symbol to define these symbols in the next version.

I tested with a number of SD shields/modules.  
You just need to edit this for your shield/module.
Code: [Select]

// SD chip select pin.
const uint8_t SD_CS_PIN = SS;


On Uno I mostly used an Adafruit data-logging shield.  It has two LEDs and I used one for an error indicator.
Code: [Select]

// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 26, 2014, 01:01 pm
I have updated the zip file attached to the first post to AnalogBinLogger20140326beta.zip.

I now determine the amount of buffering by use of the RAMEND symbol.  This allows the logger to run on ATmega chips with 2 KB or more of SRAM.

I also fixed a problem that often causes an overrun due to SD write latency to be fatal.

I would appreciate any comments on results with various SD card.  I have had the best luck with SanDisk class 4 cards.

This is the card I used in development http://www.amazon.com/dp/B007JRB0RY (http://www.amazon.com/dp/B007JRB0RY).  I tested five of these cards, two 4 GB, two 8 GB, and one 32 GB.  I have had almost no overrun errors on an Uno with these cards.

I have done some testing with this microSD http://www.amazon.com/Sandisk-MicroSDHC-Memory-Card-Adapter/dp/B000WH6H1M (http://www.amazon.com/Sandisk-MicroSDHC-Memory-Card-Adapter/dp/B000WH6H1M) It appears to work but I have no done a long reliability test.

Here is the result of a short 100 MB session with the 8 GB class 4 SanDisk  micro SD where the maximum write latency was 920 usec for a 512 byte block.
Quote

Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG01.BIN
Max block write usec: 920
Record time sec: 1993.845
Sample count: 9969226
Samples/sec: 5000.00
Overruns: 0


Here is one of the worst cards http://www.amazon.com/PNY-Optima-Class-Memory-P-SDHC4G4-EF/dp/B000L7INLU (http://www.amazon.com/PNY-Optima-Class-Memory-P-SDHC4G4-EF/dp/B000L7INLU).  I use it when I want to be sure of overruns.
Title: Re: Try this super fast analog pin logger
Post by: pito on Mar 26, 2014, 05:53 pm
Great results!  I wish I had the SdFat (or Fat16/12) running on the 8MB Ramdisk  :P
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Mar 31, 2014, 03:24 am
I tried the AnalogBinlogger program (installed the library) and it won't compile on an UNO.
I changed the "SS" to 10 for my SD card chip select for my SEEED STUDIO SD CARD shield.
I have a 1 Mb Sandisk formated with SDFAT16. Attached is the verbose compiler output pasted into WORD.
I read the Readme file and didn't really see anything that looked like configuration instructions.
Can anybody tell me what I overlooked ?
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 31, 2014, 03:38 am
bintocsv is a PC program not an Arduino library.  Remove the bintocsv folder from  Arduino libraries.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Mar 31, 2014, 04:01 am
Thanks !   That eliminated the compile errors. I moved the bintocsv folder to my Arduino reference data folder. I assume I need to remove the SD card from my reader, plug it into my PC, open a command line and then execute the command line command:
bintocsv binFile csvFile  ?  What's the easiest way to do it ? Should I copy the bin file(s) from the SD card to the bintocsv folder and execute the command as you entered it or do I need a path for anything? (windows 7)
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 31, 2014, 04:12 am
I tend to put the bintocsv.exe in the folder with the .BIN file.

You could also use a path to another location where bintocsv.exe is located.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Mar 31, 2014, 04:18 am
Thanks. I'll do it your way.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Apr 12, 2014, 09:47 pm
Quote
Place the AnalogIsrLogger folder in your sketchbook folder. 


I found this instruction in the readme file but didn't see any folder with that spelling . Is this important ?

Also, regarding these instructions:
Quote
You must edit the configuration constants at the beginning of the program
to set the sample pins, sample rate, and other configuration values.


Which file do I edit with my C++ editor ?

Quote

Initially the program is setup to log the first five analog pins at 5000
samples per second.  Change these values to suit your needs.

See RateTable.txt for maximum allowed sample rates vs pin count and ADC clock
frequency.
The program has four commands:
c - convert file to CSV
d - dump data to Serial
e - overrun error details
r - record ADC data


I thought the bin to csv had to be performed using the command line instructions in the readme file .
So when you run the program it doesn't start recording (saving to SD) until you press "r" ?
The other two "d" & "e" make total sense.
Does the above mean I can perform any of those four commands after I start running the program ?
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 12, 2014, 10:54 pm
Quote

Quote
Place the AnalogIsrLogger folder in your sketchbook folder. 


I found this instruction in the readme file but didn't see any folder with that spelling . Is this important ?


I should have just said sketchbook I guess.  I didn't say the Arduino folder since you can select the folder to use.
Quote
The Arduino environment uses the concept of a sketchbook: a standard place to store your programs (or sketches). The sketches in your sketchbook can be opened from the File > Sketchbook menu or from the Open button on the toolbar. The first time you run the Arduino software, it will automatically create a directory for your sketchbook. You can view or change the location of the sketchbook location from with the Preferences dialog. 


Quote
Which file do I edit with my C++ editor ?

You edit AnalogBinLogger.ino.  for example to select the set of analog pins edit this area:
Code: [Select]

//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------


Quote
Does the above mean I can perform any of those four commands after I start running the program ?


Yes just type the letter corresponding to the command you want run.

Quote
I thought the bin to csv had to be performed using the command line instructions in the readme file .

It takes a long time to convert large binary files on the Arduino so the PC/Mac command line program is often required.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Apr 12, 2014, 10:58 pm
THANKS !
I found it .
Code: [Select]
//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers can be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 5000;  // Must be 0.25 or greater.

// The interval between samples in seconds, SAMPLE_INTERVAL, can be set to a
// constant instead of being calculated from SAMPLE_RATE.  SAMPLE_RATE is not
// used in the code below.  For example, setting SAMPLE_INTERVAL = 2.0e-4
// will result in a 200 microsecond sample interval.
const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;

// Setting ROUND_SAMPLE_INTERVAL non-zero will cause the sample interval to
// be rounded to a a multiple of the ADC clock period and will reduce sample
// time jitter.
#define ROUND_SAMPLE_INTERVAL 1
//------------------------------------------------------------------------------
// ADC clock rate.
// The ADC clock rate is normally calculated from the pin count and sample
// interval.  The calculation attempts to use the lowest possible ADC clock
// rate.
//
// You can select an ADC clock rate by defining the symbol ADC_PRESCALER to
// one of these values.  You must choose an appropriate ADC clock rate for
// your sample interval.
// #define ADC_PRESCALER 7 // F_CPU/128 125 kHz on an Uno
// #define ADC_PRESCALER 6 // F_CPU/64  250 kHz on an Uno
// #define ADC_PRESCALER 5 // F_CPU/32  500 kHz on an Uno
// #define ADC_PRESCALER 4 // F_CPU/16 1000 kHz on an Uno
// #define ADC_PRESCALER 3 // F_CPU/8  2000 kHz on an Uno (8-bit mode only)
//------------------------------------------------------------------------------
// Reference voltage.  See the processor data-sheet for reference details.
// uint8_t const ADC_REF = 0; // External Reference AREF pin.
uint8_t const ADC_REF = (1 << REFS0);  // Vcc Reference.
// uint8_t const ADC_REF = (1 << REFS1);  // Internal 1.1 (only 644 1284P Mega)
// uint8_t const ADC_REF = (1 << REFS1) | (1 << REFS0);  // Internal 1.1 or 2.56
//------------------------------------------------------------------------------
// File definitions.
//
// Maximum file size in blocks.
// The program creates a contiguous file with FILE_BLOCK_COUNT 512 byte blocks.
// This file is flash erased using special SD commands.  The file will be
// truncated if logging is stopped early.
const uint32_t FILE_BLOCK_COUNT = 256000;

// log file base name.  Must be six characters or less.
#define FILE_BASE_NAME "ANALOG"

// Set RECORD_EIGHT_BITS non-zero to only record the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 0
//------------------------------------------------------------------------------
// Pin definitions.
//
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;
 
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Apr 13, 2014, 01:42 am
I can't seem to get the program to run. It compiles fine but when I type in "r" in the serial port and hit enter, the error led
comes on and I get a "sample rate too fast" error
I attached the sketch renamed as MyAnalogBinLogger .
Can you help me configure it to sample A0 at 40,000 samples per second ?
I don't know what to select for ADC clock or SAMPLE_INTERVAL.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Apr 13, 2014, 02:24 am
Got it to work by changing the ADC clock rate to 1000 hz . What does 8-bit mode mean for the 2000 hZ clock rate ?
see attached
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 13, 2014, 01:55 pm
Quote
Can you help me configure it to sample A0 at 40,000 samples per second ?


First you need an SD card that has very low write latency.

I found this card works very well http://www.amazon.com/SanDisk-Class-Flash-Memory-SDSDB-004G-AFFP/dp/B007JRB0RY (http://www.amazon.com/SanDisk-Class-Flash-Memory-SDSDB-004G-AFFP/dp/B007JRB0RY).

Change only these values.
Code: [Select]
//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 40000;  // Must be 0.25 or greater.

Here is serial output for recording about five minutes of data.  The proper SDHC card is extremely important.
Quote

Sample pins: 0
ADC bits: 10
ADC clock kHz: 1000
Sample Rate: 40000.00
Sample interval usec: 25.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG01.BIN
Max block write usec: 1112
Record time sec: 304.766
Sample count: 12190600
Samples/sec: 39999.87
Overruns: 0
Done


Quote

What does 8-bit mode mean for the 2000 hZ clock rate ?

8-bit mode only records the high 8-bits of each ADC  reading .  At 2000 kHz ADC clock rate the ADC only provides about 7.5 significant bits. see the ADC_ENOB.PNG file.  This reduces the amount of date written so it is easier for SD cards to handle the data rate.

I didn't include 8-bit mode in the max data rate table.  Here is a version with 8-bit mode.

Code: [Select]


ADC clock kHz 8-bit only
125 250 500 1000 2000
pins
1 7692 14286 25000 40000 57143
2 3810 6667 11111 16667 22222
3 2572 4790 8421 13559 19512
4 1942 3636 6452 10526 15385
5 1559 2930 5229 8602 12698
6 1303 2454 4396 7273 10811
7 1119 2111 3791 6299 9412
8 980 1852 3333 5556 8333
9 872 1649 2974 4969 7477
10 786 1487 2685 4494 6780
11 715 1354 2446 4103 6202
12 656 1242 2247 3774 5714
13 606 1148 2078 3493 5298
14 563 1067 1932 3252 4938
15 525 996 1806 3042 4624
16 493 935 1695 2857 4348

Here is serial monitor output for a short recording with an 18 microsecond sample interval.
Quote

Sample pins: 0
ADC bits: 8
ADC clock kHz: 2000
Sample Rate: 55555.55
Sample interval usec: 18.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG02.BIN
Max block write usec: 1304
Record time sec: 28.103
Sample count: 1561189
Samples/sec: 55552.39
Overruns: 0


Title: Re: Try this super fast analog pin logger
Post by: BrownSuedeShoes on Apr 24, 2014, 08:24 pm
I am relatively new to arduino, but I am trying to get this code working on my Yun. I am trying to run your code as is, but I am getting a number of errors. Can you help clear up what I am doing wrong?

Code: [Select]
Arduino: 1.5.6-r2 (Windows 7), Board: "Arduino Yún"

AnalogBinLogger:143: error: 'SdFat' does not name a type
AnalogBinLogger:145: error: 'SdBaseFile' does not name a type
AnalogBinLogger.ino: In function 'void error_P(const char*)':
AnalogBinLogger:251: error: 'sd' was not declared in this scope
AnalogBinLogger.ino: In function 'void binaryToCsv()':
AnalogBinLogger:416: error: 'SdFile' was not declared in this scope
AnalogBinLogger:416: error: expected `;' before 'csvFile'
AnalogBinLogger:418: error: 'BufferedWriter' was not declared in this scope
AnalogBinLogger:418: error: expected `;' before 'bw'
AnalogBinLogger:420: error: 'binFile' was not declared in this scope
AnalogBinLogger:424: error: 'binFile' was not declared in this scope
AnalogBinLogger:430: error: 'csvFile' was not declared in this scope
AnalogBinLogger:430: error: 'O_WRITE' was not declared in this scope
AnalogBinLogger:430: error: 'O_CREAT' was not declared in this scope
AnalogBinLogger:430: error: 'O_TRUNC' was not declared in this scope
AnalogBinLogger:438: error: 'bw' was not declared in this scope
AnalogBinLogger:438: error: 'csvFile' was not declared in this scope
AnalogBinLogger.ino: In function 'void checkOverrun()':
AnalogBinLogger:497: error: 'binFile' was not declared in this scope
AnalogBinLogger:501: error: 'binFile' was not declared in this scope
AnalogBinLogger:504: error: 'binFile' was not declared in this scope
AnalogBinLogger.ino: In function 'void dumpData()':
AnalogBinLogger:538: error: 'binFile' was not declared in this scope
AnalogBinLogger:542: error: 'binFile' was not declared in this scope
AnalogBinLogger.ino: In function 'void logData()':
AnalogBinLogger:584: error: 'sd' was not declared in this scope
AnalogBinLogger:596: error: 'sd' was not declared in this scope
AnalogBinLogger:604: error: 'binFile' was not declared in this scope
AnalogBinLogger:605: error: 'sd' was not declared in this scope
AnalogBinLogger:614: error: 'sd' was not declared in this scope
AnalogBinLogger.ino: In function 'void setup()':
AnalogBinLogger:752: error: 'FreeRam' was not declared in this scope
AnalogBinLogger:755: error: 'sd' was not declared in this scope
AnalogBinLogger:755: error: 'SPI_FULL_SPEED' was not declared in this scope

  This report would have more information with
  "Show verbose output during compilation"
  enabled in File > Preferences.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Apr 24, 2014, 08:29 pm
Read Reply#5 & 6 on this post. Explains about where BIN2CSV program should be located. (can't be in same directory as library)
Title: Re: Try this super fast analog pin logger
Post by: Turtle9er on Apr 28, 2014, 10:16 pm
Hi,

I was wondering if this sketch can be used on a Teensy 3.1. did a quick test, however a few errors came up with respect to Too Little Ram and QUEUE_DIM, which is due to ram error not letting those be defined. If it is a easy fix to get this working on a teensy, that would be awesome. Thanks.
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 28, 2014, 11:55 pm
Quote
I was wondering if this sketch can be used on a Teensy 3.1 .... If it is a easy fix to get this working on a teensy, that would be awesome. Thanks.


The logger is closely associated with AVR so it would require a lot of mods. 

I have no interest in porting this program to ARM since an RTOS based solution would be better on ARM.
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on May 05, 2014, 06:58 am
Attached is a sample of the toneAC library toneAC DEMO playing a whistle followed by the classic "Shave and a haircut, two bits."
melody from many many years back (probably 1800s western america era).
Title: Re: Try this super fast analog pin logger
Post by: kitkadbury on Jun 25, 2014, 04:46 am
A newbie in arduino here. I'm trying to do datalogging and print the ADC values onto a CSV file. I've configured A2 as the sample pin. On the serial monitor, the values are consistent, however on the excel file, there are gibberish characters and inconsistent jump in the values. Could someone help me please?  
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Jun 25, 2014, 04:50 am
No.
We can't see your code.
Also we don't know how you are "  trying to print the values into a CSV file".
Title: Re: Try this super fast analog pin logger
Post by: G_46 on Jun 27, 2014, 12:38 pm
Hi All,
thanks a lot to Fat16lib to share this with us. I am using the AnalogBinLogger20140326beta version on a Arduino Mega 2560 + Arduino Wireless SD Shield + Panasonic Micro SD HC I (class 10). Please note that I am new to Arduino and to C code too  :smiley-red:.
My aim is to log 4 analog pin at 200 Hz and the system must run straight after the the user will switch on the board (it is a DAQ mounted on a race car), without computer connected (so no serial monitor and no keyboard to give input to the program). Logging periods of my application are between 4 and 6 minutes

So I will really appreciate if you can help with the followings

1. I wish to modify the code in a way that the time is recorded too. [this is very important for applications that need to post process the data in the time or frequency domain. This will also give an idea the user if the logging frequency is stable or not, again this is very important if you FFT the logged data]

2. I wish to modify the code in a way that the data logging will start when the user will switch on the board and will finish when the user will switch the board off. I know that can cause some issue as the board is switched off while the record is going on, but adding a button to start stop is really the last of the option I would like to use. Perhaps the transition from .BIN to .CVS could be done later in the post processing?

I will really appreciate any help with these two question.

Thanks in advance

G
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Jun 27, 2014, 03:38 pm
Read the previous posts . The "transistion " as you call it is called "file conversion" from bin to csv. If you don't do it immediately after stopping the recording then you have to use a command window  and command line  commands, which is much more complicated. I don't think you can use this program to log timestamps. It is what it is. It logs analog data. If that's not enough for you then go write your own program.  You don't really need to log timestamps because the ADC clock speed that you set tells you what the sampling time is. Knowing the sampling time means you can create a column in Excel and fill it with the value of the sampling time added to each cell, so if the sampling time was 25uS, the first cell would contain 0. The second cell in that column would contain"25", the 3rd, "50 " and so on. When you plot the data column with respect to the column containing those values,
you are in fact plotting the analog data with respect to time in uS, are you not ?

 Read the instructions. You do not understand how to use this program because you have not read any instructions. This program does not print to CSV.  Get that idea out of your head. Secondly, the program interface is the serial monitor. The command menu is printed to the serial monitor.

Quote
The program has four commands:

c - convert file to CSV
d - dump data to Serial
e - overrun error details
r - record ADC data

All commands can be terminated by entering a character from the serial monitor.





If you are getting jibberish characters it is because your baud rate is wrong. Post a screenshot of your serial monitor screen. Also post an explanation of how you are using the program. so we can verify that you actually know what your doing . Post the explanation in the following format:
STEP-1: [DO THIS]
STEP-2 [DO THIS]
etc. .


etc

You have posted absolutely nothing that indicates you have the slightest clue how to use this program. You can see my posts with the excel plots and I had no problem using it. The fact that you state this:

Quote
I'm trying to do datalogging and print the ADC values onto a CSV file.  


Means you don't understand how this program works. It is a BIN LOGGER .That is even in the name of the sketch.
What does that mean ? It means it is NOT A CSV LOGGER. or it would be named AnalogCSVLogger.   It creates BIN files which must be converted to CSV files. You can do the conversion after you stop recording or you can do it later using the command window. The first method is easer.

In the future, if a program doesn't work, try reading the Readme file (see attaached)
Title: Re: Try this super fast analog pin logger
Post by: G_46 on Jun 28, 2014, 10:06 am

I don't think you can use this program to log timestamps. It is what it is. It logs analog data. If that's not enough for you then go write your own program. 


See Raschemmel, people like you should be banned from the forum. This is not a place to discuss about commercial software where you cannot change anything, this forum is for people sharing knowledge and to interact together to change and improve open source software. Your post is full of complains and shows your disrespectful attitude with people who are newbies.


You don't really need to log timestamps because the ADC clock speed that you set tells you what the sampling time is.  


I had already problems with sampling time from the ADC not being constant, so that s one of the reasons I am asking to log the time, for checking purposes.  If you are not able to suggest a constructive chance to the program to achieve that , then do not answer, we do not need your complains, be respectful!

And remember this is a place where people ask "politely" for help, want to find solutions and discuss different approaches and techniques, there is no space for your negative attitude, shut the f. up!

G
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Jun 28, 2014, 10:42 am
I take it that means you finally read the Readme file ?

" Your  welcome"
If anyone should be banned from the forum it's people who don't read Readme files.
I respect people who take the time to do their homework before crying "help".
Good luck loggong timestamps with the BinLogger.
FYI,
The ADC clock is accurate . If you don't believe that I dare you to post your code so we can see the proof.
BTW,
Open source doesn't mean if the software doesn't do what you want you can ask someone ELSE to change it for you.
It means YOU can can change it.  If you don't like the way it works then write your own code.
Why is it disrespectful to tell you that the reason you are getting gibberish in your Excel file is that you didn't read the Readme file and
you think the program logs ASCI to a CSV file in realtime when actually it logs BIN files (hence the name BinLogger) which must be converted to CSV AFTER the logging is complete.
Title: Re: Try this super fast analog pin logger
Post by: Matoo on Jul 11, 2014, 09:56 am
Hi, im a newbie here and im trying to interface the AnalogBinLogger code with a Nokia 5110 LCD and an SD card module. However, i keep getting these errors.

error: createContiguous failed
SD errorCode: 0X19,0X0

I tested the SD card using the CardInfo code and it works fine..
Can somebody help me please?
Thanks in advance!
Title: Re: Try this super fast analog pin logger
Post by: tharekmohtar on Aug 19, 2014, 09:55 pm
Hello guys,

I already try the code and it works like a charm. I was trying to modify it because I want to start and end the acquisition of data remotely. I bought a cheap RF transmitter and receiver (https://www.sparkfun.com/products/10532 (https://www.sparkfun.com/products/10532) and https://www.sparkfun.com/products/retired/8946 (https://www.sparkfun.com/products/retired/8946))  and downloaded the virtualwire library (http://www.airspayce.com/mikem/arduino/VirtualWire/ (http://www.airspayce.com/mikem/arduino/VirtualWire/)). To start the acquisition I modified the code like this:

Code: [Select]
void setup(void) {
 if (ERROR_LED_PIN >= 0) {
   pinMode(ERROR_LED_PIN, OUTPUT);
 }
 Serial.begin(9600);
 
 vw_setup(2000);             // Bits per sec
 vw_set_rx_pin(8) ;
 vw_rx_start();              // Start the receiver
 
 // Read the first sample pin to init the ADC.
 analogRead(PIN_LIST[0]);
 
 Serial.print(F("FreeRam: "));
 Serial.println(FreeRam());

 // initialize file system.
 if (!sd.begin(SD_CS_PIN, SPI_FULL_SPEED)) {
   sd.initErrorPrint();
   fatalBlink();
 }
}
//------------------------------------------------------------------------------
void loop(void) {
 Serial.println();
 Serial.println(F("type:"));
 Serial.println(F("c - convert file to CSV"));
 Serial.println(F("d - dump data to Serial"));  
 Serial.println(F("e - overrun error details"));
 Serial.println(F("r - record ADC data"));

 while(!vw_get_message(message, &msgLength)) {}
 if (ERROR_LED_PIN >= 0) {
   digitalWrite(ERROR_LED_PIN, LOW);
 }
 
 if (message[0] == 'c')
 {
   binaryToCsv();
 }
 else if (message[0] == 'd')
 {
   dumpData();
 }
 else if (message[0] == 'e')
 {
   checkOverrun();
 }
 else if (message[0] == 'r')
 {
   logData();
 }
 else
 {
   Serial.println(F("Invalid entry"));
 }
}


So when i send the letter r from my other arduino  the one with the sd card starts recording. The problem is when I try to stop it. To do so I modified the logdata function:

Code: [Select]
   if (vw_get_message(message, &msgLength)) {
     // stop ISR calls
     adcStop();
     if (isrBuf != 0 && isrBuf->count >= PIN_COUNT) {
       // Truncate to last complete sample.
       isrBuf->count = PIN_COUNT*(isrBuf->count/PIN_COUNT);
       // put buffer in full queue
       fullQueue[fullHead] = isrBuf;
       fullHead = queueNext(fullHead);
       isrBuf = 0;
     }
     if (fullHead == fullTail) break;
   }


But the former does not work. I read in the virtualwire documentation that it uses timer1 and I think that is why my code is not working but I don't know how to fix it.

Any advice?

Thanks in advance to you all
Title: Re: Try this super fast analog pin logger
Post by: raschemmel on Aug 20, 2014, 12:03 am
There is only one way to stop the recording. Put the cursor in the serial monitor entry window and hit any key.
The serial monitor menu prints a promot to that effect when it starts the recording . The menu is only viewable on the serial monitor.
After stopping the recording, if you do not endter the letter command to convert the file to CSV then you will have to convert it from binary later using the dos command window using the command line commands for converting it. It is MUCH more convenient to do the conversion immediately after stopping the recording.

READ THE README FILE . EVERYTHING IS IN THERE.
Title: Re: Try this super fast analog pin logger
Post by: tharekmohtar on Aug 20, 2014, 02:08 pm
Thanks for your answer,

I am trying to to start and stop the acquisition using a RF transmitter and receiver. My question had nothing to do with the conversion between bin to csv. I was asking how to solve the conflict between virtualwire and the analogbinlogger since, I think, they both use timer1.

Thanks again  
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Aug 20, 2014, 03:04 pm
Edit: I suspect using VirtualWire will cause this logger to fail, even if you modify it to use a different timer.  VirtualWire takes too long in its interrupt routine.

Timer1 is used to start the ADC conversion.  Only timer0 or timer1 can be used to trigger the ADC but timer0 is used for mills() and micros().

Timer2 is free but a mod to the VirtualWire library would be required.  This is probably not too difficult since VirtualWire runs on ATtiny85 using timer0 which is 8-bits.  I believe timer2 has all the required features.

If you can live with logging at 500 Hz or less, the new SdFat has a logger that does not use timer1, the LowLatencyLogger example.

The new library is here https://github.com/greiman/SdFat (https://github.com/greiman/SdFat).

Using VirtialWire with any fast logger may cause problems.
Title: Re: Try this super fast analog pin logger
Post by: tharekmohtar on Aug 20, 2014, 05:08 pm
Thank you so much for your answer. I need 5kHz so I will try to change the virtualwire library.

Best regards.
Title: Re: Try this super fast analog pin logger
Post by: koksy123 on Nov 17, 2014, 08:20 pm
Hi fat16lib,

First of all, I really appreciate your work and I just wanted to let you know that it helped me a lot. I downloaded your AnalogIsrLogger20121219 zip.file(I know it is the older version) and currently working on logging the Analog pin.

Detailed Specifications
1) Arduino Uno
2) 32GB SanDisk SDHC card
3) Attempting to log 250Hz, 1V sine wave into SDcard. (using a Digilent Explorer board to generate signal input to arduino)

So far, the only thing that i changed in your code is setting the SD chip select pin from SS to 4.

// SD chip select pin
const uint8_t chipSelect = 4;

I was able to access the menu shown in "Menu" snippet in the attachment and was able to log the data and covert the binary to a text file.

CURRENT ISSUE:
When i try to feed analog data to the Arduino into port A0, i do not get the results i want. The code does not seem to detect any input signal, the interrupt does not seem to be working as well. The result is a constant value that does not change even when i manipulate the frequency or amplitude of the analog signal. Refer to the "Result" snippet at the attached.

Can someone please help me out with this? I do not understand as to why this is happening.





Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Nov 18, 2014, 01:28 am
Quote
AnalogIsrLogger20121219 zip(I know it is the older version)
Please download the latest SdFat from GitHub and try the current version of AnalogBinLogger.

https://github.com/greiman/SdFat (https://github.com/greiman/SdFat)

The example is here https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger (https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger)
Title: Re: Try this super fast analog pin logger
Post by: Gerry48 on Nov 18, 2014, 06:11 pm
I'm doing the same thing, starting & stopping analogbinlogger with a RF transmitter/receiver.  I don't use virtualwire.  Just connect RF receiver output to one of the digital input pins.  Check this pin for HI or LO signal.




Thanks for your answer,

I am trying to to start and stop the acquisition using a RF transmitter and receiver. My question had nothing to do with the conversion between bin to csv. I was asking how to solve the conflict between virtualwire and the analogbinlogger since, I think, they both use timer1.

Thanks again 
Title: Re: Try this super fast analog pin logger
Post by: golinaveenkumar on Nov 19, 2014, 07:12 pm
Hi fat16lib.
I want to record data in kHz from arduino to SDcard. I have 2GB SD card FAT16, and SDcard holder.
I downloaded your AnalogBinLogger20140326beta.zip(1.88MB) file. i tried for two days following your instructions and changing files and folders here and there and fully messed up. i just want a fresh start.
I uninstalled and installed Arduino 1.0.5-r3.

Kindly suggest the procedure steps after unzipping the file(i.e, what files to keep where). i have read your 'read me' file. sorry to say that i got confused. plz tell me where to keep SdFat.h, SdFat.cpp, BufferedWriter.h, BufferedWriter.cpp and remaining files if any.

I read the previous posts but didn't make out. kindly give the steps to follow

Thank you
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Nov 19, 2014, 08:30 pm
I have removed the beta version of this program.  Please download the newest SdFat from GitHub https://github.com/greiman/SdFat (https://github.com/greiman/SdFat).

AnalogBinLogger is now a SdFat example https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger (https://github.com/greiman/SdFat/tree/master/SdFat/examples/AnalogBinLogger).

Also see this directory https://github.com/greiman/SdFat/tree/master/AnalogBinLoggerExtras (https://github.com/greiman/SdFat/tree/master/AnalogBinLoggerExtras)

See the SdFat readme file and the html documentation for installing SdFat.

Quote
You must manually install SdFat by copying the SdFat folder from the download package to the Arduino libraries folder in you sketch book.

See the Manual installation section of this guide http://arduino.cc/en/Guide/Libraries (http://arduino.cc/en/Guide/Libraries)

Title: Re: Try this super fast analog pin logger
Post by: koksy123 on Nov 19, 2014, 10:57 pm
Hi Fat16lib,

I finally got it working, using the 2014beta version!! So i was able to sample a 250Hz wave and reconstruct the signal which is then played in MATLAB. It sounds pretty good, almost perfect! Thought it would be nice to let you know!

Questions
1) I realized once i set my SAMPLE_RATE to be greater than 8000Hz, I receive an error saying "Sample rate too high". Why is that? Am i bounded by the limitations of the SD card write latency?

2) Also, in the readme, it says

"If your SD card has a long write latency, it may be necessary to use slower sample rates."

How would you measure the optimal sample rate to use? I would think, a higher sample rate would always be better.But then again i'm a newbie!

Reminder:
Detailed Specifications
1) Arduino Uno
2) 32GB SanDisk SDHC card
3) Attempting to log 250Hz, 1V sine wave into SDcard. (using a Digilent Explorer board to generate signal input to arduino)


Sorry for these questions, I'm new, but i'm willing to learn!! Let me know if you want any data/testing from me. I would gladly obliged :)



Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Nov 20, 2014, 01:51 pm
Quote
I realized once i set my SAMPLE_RATE to be greater than 8000Hz, I receive an error saying "Sample rate too high". Why is that? Am i bounded by the limitations of the SD card write latency?
Many of your questions are covered in the folder I asked you to look at https://github.com/greiman/SdFat-beta/tree/master/AnalogBinLoggerExtras (https://github.com/greiman/SdFat-beta/tree/master/AnalogBinLoggerExtras).

The maximum possible sample rate depends on how many pins you are logging and the ADC clock frequency. See RateTable.txt in in the the above folder.  Also see the readme.txt and  AdcErrorStudy.txt.


If your SD card has latency problems, you will get data overrun errors.  A Mega will eliminate SD latency problems since it has more RAM.

If you log only one pin, you  may be able to log at 40,000 Hz.

Try the following:

Code: [Select]

//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 40000;  // Must be 0.25 or greater.

Here is a short run I did on an Uno at 40,000 Hz logging Analog pin zero.
Quote
Sample pins: 0
ADC bits: 10
ADC clock kHz: 1000
Sample Rate: 40000.00
Sample interval usec: 25.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG04.BIN
Max block write usec: 1116
Record time sec: 77.307
Sample count: 3092251
Samples/sec: 39999.62
Overruns: 0
You can log slightly faster if you select 8-bit mode.
Code: [Select]

// Set RECORD_EIGHT_BITS non-zero to record only the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 1


Here is a run with 18 μs between samples, about 55,555 Hz.

Quote
Sample pins: 0
ADC bits: 8
ADC clock kHz: 2000
Sample Rate: 55555.55
Sample interval usec: 18.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG07.BIN
Max block write usec: 1304
Record time sec: 142.443
Sample count: 7913444
Samples/sec: 55555.16
Overruns: 0
Done

Title: Re: Try this super fast analog pin logger
Post by: golinaveenkumar on Nov 21, 2014, 02:33 am
Thanks fat16lib.
I could able to record 10k sampling for three which is my requirement.
I have one more question, whether the samples can be time stamped by any means. If yes, it is by software or any extra hardware to be fitted.

I need like
Time             Data1            Data2            Data3
0.00001         1023                 0                 1023
0.00002         1023                 0                 1023
.....
.....
.....
......
 It will be very helpful to my project.
Thank you.
Title: Re: Try this super fast analog pin logger
Post by: golinaveenkumar on Nov 21, 2014, 03:50 am
Hi fat16lib,
I got that time stamping from excel(little gimmick). can get the real time clock to my arduino. But it is in I2C mode.It gets delayed na

Can i get

Date                   Time               Data1
21/11/2014    14:24:00001         1023
21/11/2014    14:24:00002         1023
21/11/2014    14:24:00003         1023
....
....
....
....
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Nov 21, 2014, 03:53 am
Quote
I have one more question, whether the samples can be time stamped by any means
The ADC is triggered by timer 1 at intervals that are exactly the same number of cpu cycles.

The time is totally hardware controlled.  The ADC is started on the edge of CPU clock and read out in an ADC done ISR.

If you are logging at 10,000 Hz, the interval is exactly 1,600 cycles for a 16 MHz CPU.  So your data points have a 100 μs interval with a jitter of a fraction of a CPU cycle.

There is no point in writing the time to the file.  I do insert overrun error markers if data is lost due to SD
latency problems.

Your RTC time is worthless just at this level.  Looks like you excel interval is off by a factor of ten.  should be 0.0000, 0.0001, 0.0002, ....
Title: Question on the SDcard
Post by: koksy123 on Jan 31, 2015, 11:06 pm
Hi fat16lib!,

I finally got it to work completely and I now want to transfer the data wirelessly to the computer through WiFi.

Is it possible that I am able to write & read from the binary file as data is being logged? I want to transfer the read data over the internet through WiFi.

thank you so much again!

Best,
Dennis
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Feb 01, 2015, 12:09 am
Quote
Is it possible that I am able to write & read from the binary file as data is being logged?
No.
Title: Re: Try this super fast analog pin logger
Post by: koksy123 on Feb 09, 2015, 06:09 pm
No.
Instead of logging it into the SDcard, would it be possible i sent the data over the internet? If so, how would you go about it? Sorry I am new and still a learning to be a decent technologist. Thanks.
Title: Bin file not working
Post by: koksy123 on Feb 18, 2015, 06:22 am
Hi Fat16lib,

Again, I was able to use Matlab to reconstruct a Sinewave and using a FFT function to prove the frequency of the sine wave. (using the csv data)

HOWEVER, when i am used the .bin data, and import the raw data into Audacity, I do not hear anything or observe a sine wave waveform even with the right settings. Why is that?
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Feb 18, 2015, 04:46 pm
Quote
HOWEVER, when i am used the .bin data, and import the raw data into Audacity, I do not hear anything or observe a sine wave waveform even with the right settings. Why is that?
The bin file has a record structure for recording data in 512 byte blocks.  You need a program on your PC/Mac to convert the file to the needed format.  This (https://github.com/greiman/SdFat/tree/master/AnalogBinLoggerExtras/bintocsv) example shows how to read the file on a PC/Mac.

The example converts the data to a csv file but you can write the data in any format by modifying the program.
Title: Re: Try this super fast analog pin logger
Post by: koksy123 on Mar 07, 2015, 12:46 am
Quote
The bin file has a record structure for recording data in 512 byte blocks.
So what would be the encoding structure of this raw data in the .bin file? Please share light as to how recording byte blocks data may differ in bytes data in terms of encoding, and why i can't just simply play the .bin file in MATLAB without converting the file if using a record structure of 512 byte blocks. Thanks you, and would appreciate a response.



Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 08, 2015, 12:15 am
Quote
why i can't just simply play the .bin file in MATLAB without converting the file if using a record structure of 512 byte blocks.
The record structure means there is other stuff in the 512 byte block and some unused space.  You should read the comment at the start of the program.
Code: [Select]

 * Each 512 byte data block in the file has a four byte header followed by up
 * to 508 bytes of data. (508 values in 8-bit mode or 254 values in 10-bit mode)
 * Each block contains an integral number of samples with unused space at the
 * end of the block.


Title: Re: Try this super fast analog pin logger
Post by: koksy123 on Mar 08, 2015, 07:14 am
The record structure means there is other stuff in the 512 byte block and some unused space.  You should read the comment at the start of the program.
Yes I did read it. The 4 byte header that you mentioned, are they the headers in the picture I've attached?  (the 'Interval' and 'pin'), if not, what are the headers?

Also, why are there unused space? Sorry for the questions, but this is really helping me learn.
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 08, 2015, 02:41 pm
Quote
The 4 byte header that you mentioned, are they the headers in the picture I've attached?  (the 'Interval' and 'pin'), if not, what are the headers?
Look at AnalgogBinLogger.h.
Code: [Select]

//------------------------------------------------------------------------------
// First block of file.
struct metadata_t {
  unsigned long  adcFrequency;     // ADC clock frequency
  unsigned long  cpuFrequency;     // CPU clock frequency
  unsigned long  sampleInterval;   // Sample interval in CPU cycles.
  unsigned long  recordEightBits;  // Size of ADC values, nonzero for 8-bits.
  unsigned long  pinCount;         // Number of analog pins in a sample.
  unsigned long  pinNumber[123];   // List of pin numbers in a sample.
};
//------------------------------------------------------------------------------
// Data block for 8-bit ADC mode.
const size_t DATA_DIM8 = 508;
struct block8_t {
  unsigned short count;    // count of data values
  unsigned short overrun;  // count of overruns since last block
  unsigned char  data[DATA_DIM8];
};
//------------------------------------------------------------------------------
// Data block for 10-bit ADC mode.
const size_t DATA_DIM16 = 254;
struct block16_t {
  unsigned short count;    // count of data values
  unsigned short overrun;  // count of overruns since last block
  unsigned short data[DATA_DIM16];
};


The first block is type metadata_t.  The rest of the blocks are type block8_t or block16_t.

The data block header has a count of values in data[] and a count of overruns (missed samples) since the last block. 

The block will contain up to SAMPLES_PER_BLOCK samples. A sample consists of PIN_COUNT values. 

Overruns happen when no block buffer is available due to SD write latency.

To understand this please read the binaryToCsv() function.
Code: [Select]

//------------------------------------------------------------------------------
// Convert binary file to csv file.
void binaryToCsv() {
  uint8_t lastPct = 0;
  block_t buf;
  metadata_t* pm;
  uint32_t t0 = millis();
  char csvName[13];
  StdioStream csvStream;

  if (!binFile.isOpen()) {
    Serial.println(F("No current binary file"));
    return;
  }
  binFile.rewind();
  if (!binFile.read(&buf , 512) == 512) {
    error("Read metadata failed");
  }
  // Create a new csv file.
  strcpy(csvName, binName);
  strcpy(&csvName[BASE_NAME_SIZE + 3], "csv");

  if (!csvStream.fopen(csvName, "w")) {
    error("open csvStream failed");
  }
  Serial.println();
  Serial.print(F("Writing: "));
  Serial.print(csvName);
  Serial.println(F(" - type any character to stop"));
  pm = (metadata_t*)&buf;
  csvStream.print(F("Interval,"));
  float intervalMicros = 1.0e6*pm->sampleInterval/(float)pm->cpuFrequency;
  csvStream.print(intervalMicros, 4);
  csvStream.println(F(",usec"));
  for (uint8_t i = 0; i < pm->pinCount; i++) {
    if (i) {
      csvStream.putc(',');
    }
    csvStream.print(F("pin"));
    csvStream.print(pm->pinNumber[i]);
  }
  csvStream.println();
  uint32_t tPct = millis();
  while (!Serial.available() && binFile.read(&buf, 512) == 512) {
    uint16_t i;
    if (buf.count == 0) {
      break;
    }
    if (buf.overrun) {
      csvStream.print(F("OVERRUN,"));
      csvStream.println(buf.overrun);
    }
    for (uint16_t j = 0; j < buf.count; j += PIN_COUNT) {
      for (uint16_t i = 0; i < PIN_COUNT; i++) {
        if (i) {
          csvStream.putc(',');
        }
        csvStream.print(buf.data[i + j]);
      }
      csvStream.println();
    }
    if ((millis() - tPct) > 1000) {
      uint8_t pct = binFile.curPosition()/(binFile.fileSize()/100);
      if (pct != lastPct) {
        tPct = millis();
        lastPct = pct;
        Serial.print(pct, DEC);
        Serial.println('%');
      }
    }
    if (Serial.available()) {
      break;
    }
  }
  csvStream.fclose();
  Serial.print(F("Done: "));
  Serial.print(0.001*(millis() - t0));
  Serial.println(F(" Seconds"));
}
Title: Re: Try this super fast analog pin logger
Post by: promavto on Mar 24, 2015, 07:45 am
Hi fat16lib!

I want to add a menu on the display using the library UTFT.
Error block the implementation of the program.
I ruled out of the program
// Buffer definitions. and applied

// Use total of five 512-byte buffers.
const uint8_t BUFFER_BLOCK_COUNT = 4;
// Dimension for queues of 512 byte SD blocks.
const uint8_t QUEUE_DIM = 8;
// Must be a power of two!

The program runs without errors.
How to apply except block?

I use Arduino Mega 2560
My project is on https://github.com/promavto/trunk
Kind regards!
promavto@ntmp.ru
Title: Re: Try this super fast analog pin logger
Post by: promavto on Mar 24, 2015, 07:49 am
Sorry, I used only  :o

// Use total of five 512-byte buffers.
const uint8_t BUFFER_BLOCK_COUNT = 4;
// Dimension for queues of 512 byte SD blocks.
const uint8_t QUEUE_DIM = 8;
// Must be a power of two!
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Mar 24, 2015, 08:00 am
have anybody tried it with arduino Mega 2560 and adafruit data logging shield  ? I get and error recognizing the card
Title: Re: Try this super fast analog pin logger
Post by: promavto on Mar 24, 2015, 08:15 am
Change
// SD chip select pin.
const uint8_t SD_CS_PIN = SS;

to Change
// SD chip select pin.
const uint8_t SD_CS_PIN = 53;


Change
if (!sd.begin(SD_CS_PIN, SPI_FULL_SPEED))
{
sd.initErrorPrint();
fatalBlink();
 }

to Change
// Initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
 // breadboards.  use SPI_FULL_SPEED for better performance.

if (!sd.begin(SD_CS_PIN, SPI_HALF_SPEED))
{
sd.initErrorPrint();
fatalBlink();
 }

Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 26, 2015, 05:04 am
I am trying to integrate SdFatLib with bluetooth LE using Adafruit nRF8001.  From what I could find, SD library use SPI_MODE_0, and the nRF8001 uses SPI_MODE_1.   

I can include the Adafruit_BLE_UART.h header, but as soon as I instantiate a BLE_UART instance, I can no longer log data using AnalogBinLogger.ino

I would really like to use this library, as it is very fast and robust (I have not  run into any overrun errors).  I've seen a post on Adafruit forums, but integration to SdFat was not solved. 

If I switch to SPI_MODE_0 to start the AnalogBinLogger, will that disconnect my connection to BLE?

Thanks,
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 26, 2015, 07:12 pm
Edit: Just noticed this.  SdFat uses MSBFIRST.
Code: [Select]
SPI.setBitOrder(LSBFIRST);

It appears that the nRF8001 uses SPI_MODE0.

This is from the Adafruit_nRF8001 library, file hal_aci_tl.cpp.
Code: [Select]

  SPI.setDataMode(SPI_MODE0);


It is not likely you can use this Bluetooth library with AnalogBinLogger because of the way the nRF8001 library uses interrupts.  The interrupt latency of the nRF8001 library is too long.

The nRF8001 also uses a very slow SPI clock, SPI_CLOCK_DIV8, which will not work for fast logging.

SdFat set the SPI bus before every access so the Bluetooth settings will not hold.  The Bluetooth library seems to only set the SPI options in the beginning.
Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 26, 2015, 08:15 pm
Hi fat16lib,

Thanks for your response!  Well I guess that's good news that SPI_MODE0 for both SPI devices.  I thought it uses SPI_MODE1 through an Adafruit post. OK, it seems like it's too difficult, if not impossible, to integrate the two libraries.   One last stab at this problem:
* I am using Arduino Mega with SD shield installed on pins 10-13.   The BLE device is installed on pins 50-53. Will using separate pins allow compatibility? Or does setting SPI bus apply to both hardware and software SPI pins?
* Do you think RedBear Lab's BLE library would be compatible?

Is a possible workaround:
* To send data between two Arduinos (one uC for BLE, one for SdFat)? I am thinking serial, but if there is a compatible one, please mention. Will this create the same conflict?
* Is it possible to record >3kHz data for 2 analog pins for 1 second using SD.h library? If so, how would you suggest going about this?
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 26, 2015, 10:13 pm
Use of Bluetooth while logging at high speed has basic problems.  Most Bluetooth shields use SPI in a way that is not compatible with high speed logging.  AVR board like the Mega have only one hardware SPI controller.

SdFat supports software SPI but it is much slower than hardware SPI.  I don't know if any Bluetooth libraries can use software SPI.

Maybe other users can help.

It would likely work if you could disable Bluetooth while logging data.  There may be an end() call in the Bluetooth library.

Logging data at high speeds requires very low time jitter.  You need to have the error in time to be comparable to the error in value.  If you want 1% accuracy at 3 kHz, the jitter in time must be no more than a few μs.  This limits what you can do in parallel with data logging.
Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 26, 2015, 11:29 pm
Thank you fat16lib. I will try ending the bluetooth connection before calling analogBinLogger, and report back.  However, I am left confused how to get the data to the phone.  I don't think I can reconnect 32000x times to send BLE packets (20bytes long for 20kB total data).

Do you know if SdFat is compatible with wire.h? I have seen a few posts regarding this compatibility. If the different SPI devices are located on different arduinos, will I still see conflict?

Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 27, 2015, 12:01 am
Dangit. There is no end() function the Adafruit_BLE_UART library.  I hope the dual arduinos approach can work...
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 27, 2015, 01:22 am
SD cards are SPI devices and can't use I2C.  I2C is also way too slow, Wire runs at 100 kHz.  SPI runs at 8 MHz, 80 times faster.

My new software SPI runs at about 2 MHz with SdFat.

I tried AnalogBinLogger with software SPI.  I logged 4 pins at 5 kHz on a Mega using an Adafruit shield that has no ISP connector, just connections to pins 10-13.  Hardware SPI on Mega is the ISP or pins 50-53.

It ran successfully with no overruns.

Quote
Sample pins: 0 1 2 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: analog02.bin
Max block write usec: 5196
Record time sec: 56.500
Sample count: 282481
Samples/sec: 4999.66
Overruns: 0
Done
A Bluetooth library could still cause problems if interrupts are disabled too long.

You can enable software SPI in SdFat by editing SdFatConfig.h and set this:
Code: [Select]
#define SD_SPI_CONFIGURATION 2
Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 27, 2015, 03:48 am
It doesn't seem the Bluetooth library will work.  I could not get it working reliably with <SD.h>.   I don't understand enough of the programming in SdFat to modify to work with the Adafruit BLE library.

I didn't mean use I2C for SD connections.  I meant connect the SPI devices on separate Arduinos and send the data between the arduinos via I2C, so there is not SPI conflict. I am hoping I2C does not have conflict with SPI.

In this configuration, master Arduino has the BLE breakout, and slave Arduino has the SD + analogBinLogger.   Communication via I2C to send data between master and slave. Master sends data to iOS app.

After logData() has been called, I would use a modified dumpData() function, to transmit the recorded data so master via i2c, instead of printing to serial.  I'm OK with I2C slowing down dumpData() loop, provided that this will work.

Madness? I think so.  Perhaps understanding how to use <SD.h> and writing analog signals using adc() is more reasonable? However this is my problem that I need solved, it seems like using a Due might work, so that I can have enough RAM for 2x 5kHz measurement @ 1 second. But then getting the BLE to play with the Due is another story...
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 27, 2015, 02:34 pm
Why can't you just use a simple Bluetooth modem like this (https://www.sparkfun.com/products/12577)?

Quote
These Bluetooth wireless modems work as a TTL serial (RX/TX) pipe, as a replacement for serial cable connections.
You just hook it to the serial pins on Arduino

I know modems like BlueSMiRF have been used with ios and Andriod.  You need to make sure you find one that works with ios.

Looks like this (http://redbearlab.com/blemini/) device works with ios and connects to Arduino serial.

The BearLab shield appears to connect to SPI.
Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 27, 2015, 07:58 pm
Bluetooth classic requires additional enrollment to Apple MFi program, or phone must be jailbroken.  These requirements will not work for this application.

I added MasterWriter/SlaveReceiver example to analogBinReader() and it was unsuccessful. Using Wire.onReceive(receiveEvent), A serial.print() statement works within receiveEvent() but if logData() is placed inside, logData() does not work from Serial port or from Wire.h call. You were right, I was not able to use I2C to call the analogBinLogger() library.

I just learned ICSP wires digital pins 11-13 to the hardware SPI. Explains why SdFat was working without declaring Software SPI. 

On the Mega, if I ran software SPI for the shield (bypassing ICSP headers) and BLE shield on 50-53 will the SPI calls still conflict? I am going to try this next approach.  post #5 suggests having run on software SPI might help these two libraries work (http://forum.arduino.cc/index.php?topic=54237.0)



Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Mar 27, 2015, 11:18 pm
This (http://redbearlab.com/blemini/) device is BLE, not Bluetooth classic.  See the ios section of the link (http://redbearlab.com/blemini/).  This device connects to the Serial pins.

Quote
You can now incorporate Bluetooth 4.0 Low Energy (BLE) technology easily in your project with our BLE Mini. BLE Mini requires only a serial port for communication so it supports all major development platforms that have UART interface including Arduino, Raspberry Pi, Netduino, BeagleBone etc.
Quote
On the Mega, if I ran software SPI for the shield (bypassing ICSP headers) and BLE shield on 50-53 will the SPI calls still conflict?
I often use an Adafruit data logging shield on a Mega with software SPI it connects to pins 10-13 and does not access the Mega hardware SPI on pins 50-53 which are also connected to the ISP pins.  The Adafruit data logging shield has no ISP connector.  There is no conflict with hardware SPI pins.

There are several problems with the Adafruit library so it may conflict with any approach to fast logging.  The Adafruit Bluetooth shield uses an interrupt and is slow so it may block reading the adc fast enough unless you can disable Adafruit Bluetooth while logging.

The reason you can't share the SPI bus with the Adafruit Bluetooth shield is that it does not set SPI options properly.  The Adafruit Bluetooth library needs to be updated to use the new Arduino 1.6x SPI library with SPI transactions.

Be aware that logging two pins at 5 kHz can't even be done with the standard analogRead() on a Mega.  I increase the ADC clock by a factor of up to four.  Any extra latency will prevent logging at the speed you want.
Title: Re: Try this super fast analog pin logger
Post by: pito on Mar 28, 2015, 10:55 am
If you are going to play with sending fast data via any serial interface (uart, BLE, SPI, I2C) always start with calculating the bandwidth (kBytes/sec) you need. Then you may simply simulate the data stream to assess, whether your link is even capable to transfer such amount of data reliably, or not.

An example - I want to stream 5000 samples per second, each sample consists of 4 ADC readings.
You may pack in four 10bit ADC values into 5 bytes when lucky, and you will not use any samples separator like new line or carriage return.

So the raw speed you need to achieve with this example is 5000 x 5 = 25kBytes per second.

Now you may try to send 25kBytes/second via your link, ie BLE or UART or whatever else, in a simple loop, to verify whether it works reliably.





Title: Re: Try this super fast analog pin logger
Post by: swca on Mar 30, 2015, 01:42 am
The BLE mini you have suggested looks pretty good. It seems like the best solution to integrate BLE with the SdFat library! Also leaves me room to add more potentially conflicting SPI devices, heh...

However, it sounds like the Adafruit BLE module is salvageable if I can connect the SD and SdFat to software SPI, and the BLE to hardware SPI, then the two will not be conflicting... is this true? Actually BLE to software SPI, and BLE to hardware SPI seems more optimized.

I am really impressed by your work, which is why I would like to use this library.


Thanks all for your responses, this is all very helpful to me!
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Apr 01, 2015, 07:27 am
I was just trying to understand the code of the datalogger (AnalogBinLogger)

just my question is about the logic that combines the timer and the "ADC done interrupt" , as I understand in ISR(ADC_vect) is performing a new conversion when the last conversion is done, so some how this will keep running according to the adc speed which can be determined by the prescaler and the CPU frequency, then the timer some how is used to get the the samples according to the sampling intervals. how is the timer " connected" to ISR(ADC_vect) .

fat16lib could you just explain how it works, the general concept without coding, the flow of the timer and the adc interrupt ?

and just to confirm: I guess timer 0 is just for Millis and Micros and timer 1 is the one that related to the sampling frequency and ish ?
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 01, 2015, 03:24 pm
Quote
fat16lib could you just explain how it works, the general concept without coding, the flow of the timer and the adc interrupt ?
The ADC conversion for the first pin is started by timer 1, not by an interrupt.  This allows the jitter in the first conversion to be less than one CPU cycle no matter what else is happening.  The ADC conversions for the remaining pins are started in the ADC done ISR.

See the AVR datasheet to see how timer 1 can trigger an ADC conversion.
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Apr 03, 2015, 03:55 am
thanks fat16lib that helped a lot,

I am trying to use some of these techniques to send data through an Ethernet shield to a web page in fixed intervals

another question fat16lib, what is the flow of writing these blocks to the memory card, I mean like, lets say for the Mega we have 13 blocks that use the RAM space, my question is: when do you write these blocks to the memory? after they all get full? and if so, while writing into the SD card, if these blocks are full, where do you store the new samples?

that would help and save me a great portion of time. I need just the flow and the idea no coding

thanks
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 03, 2015, 04:53 am
I write a block to the SD as soon as possible after it is full.
Title: Re: Try this super fast analog pin logger
Post by: Littlechap on Apr 12, 2015, 06:12 pm
Hi Fat16lib,

I was wondering if its possible to use this to find data from an mpu 6050 connected to an Uno and Adafruit data logger at a high sample rate (about 1000Hz)?

Also, whenever i try to run the analogbinlogger example, it can never locate any of the files such as SDFat.

Thanks.
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Apr 12, 2015, 07:02 pm
Quote
I was wondering if its possible to use this to find data from an mpu 6050 connected to an Uno and Adafruit data logger at a high sample rate (about 1000Hz)?
No, this logger can only log analog pins.  The mpu 6050 is an I2C device.

Quote
Also, whenever i try to run the analogbinlogger example, it can never locate any of the files such as SDFat.
The SdFat folder must be in the wrong location.  On Windows the folder need to be here
Quote
C:\Users\<user name>\Documents\Arduino\libraries\SdFat
Title: Re: Try this super fast analog pin logger
Post by: Littlechap on Apr 12, 2015, 07:38 pm
Damn.
I thought as much but worth an ask. Thanks for the speedy reply. Seems like its impossible to log the data onto an SD at 1000Hz
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Jun 11, 2015, 04:13 am
Hi @fat16lib
first of all I would like to thank you because you did a great thing to humanity.

I just wanna know how to use the: bintocsv program, I mean if I want to do the conversion using the PC not the Arduino

when I open up the bintocsv.exe, it remains for less than one second and then it disappears
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Jun 11, 2015, 03:37 pm
Quote
when I open up the bintocsv.exe, it remains for less than one second and then it disappears
See  readme.txt
Quote
The bintocsv folder contains a PC program for converting binary files to
CSV files.  I have included a executable for Windows.  Linux and Mac users
can build from the included source files.  bintocvs is a command line program.

bintocsv binFile csvFile
Here is sample output:
Quote
C:\tmp>bintocsv analog01.bin analog01.csv
pinCount: 5
Sample pins: 0 1 2 3 4
ADC clock rate: 500 kHz
Sample rate: 5000 per sec
Sample interval: 200.0000 usec
124620 ADC values read
Here is the start of analog01.csv
Quote
Interval,200.0000,usec
pin0,pin1,pin2,pin3,pin4
285,318,308,337,1023
713,570,470,422,1023
900,774,655,562,1023
975,897,801,700,1023
1004,960,896,809,1023
1012,989,949,884,1023
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Jul 21, 2015, 04:54 am
Hiii @fat16lib,  in ADC clock frequency  Versus ENOB, lets say in the case of 250 KHz ADC clock frequency, the ENOB is 9.5,

when I am converting back to volts again, should I divide by 2^ (10 ) or 2^(9.5) ?

thanks
Title: Re: Try this super fast analog pin logger
Post by: snoop911 on Aug 06, 2015, 07:13 am
Regarding the SD card writing, is this an optimized implementation of elm-chan FatFs or is it completely written from scratch for the adc logging?

What currently prevents it from recording data faster than 40 KSps for 1 channel?

For example, I've been thinking of digitizing a camera feed with a ultra fast adc, a UHS class sd card (30MB/s), and a 10Mhz SPI clock to the SD..   what would the expected data rate that could be reliably saved to the card without  dropping samples?



Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Aug 06, 2015, 01:25 pm
Quote
optimized implementation of elm-chan FatFs
No this is faster and smaller than FatFs.  It is not custom for AVR ADC so the method can be used for other  logging.

Quote
What currently prevents it from recording data faster than 40 KSps for 1 channel?
Mainly the AVR ADC speed and the overhead of servicing ADC done interrupts.  I have pushed the ADC clock as much as possible. For Uno buffering is a limit, see below.

Quote
For example, I've been thinking of digitizing a camera feed with a ultra fast adc, a UHS class sd card (30MB/s), and a 10Mhz SPI clock to the SD..   what would the expected data rate that could be reliably saved to the card without  dropping samples?
The UHS card may go slower.  The speed rating is for fast clock in SDIO mode, not SPI mode.

The big problem will be enough buffering.  Modern fast SD cards assume you have lots of buffer so there can be occasional long latency writes while the card does wear leveling or erases large flash areas.

Cards are allowed as much as a 250 ms write latency but most cards can log 40 kSps with one byte samples.  If you push the cards you will get long latencies.  The standard allows up to 250 ms

Here is the spec:
Quote
In case of High Capacity SD Memory Card, maximum length of busy is defined as 250ms for all write operation.
While the card should try to maintain that busy indication of write operation does not exceed 250ms in the case of SDXC card, if the card is not possible to maintain operations with 250ms busy, the card can indicate write busy up to 500ms including single and multiple block write in the following scenarios:
a)The last busy in any write operation up to 500ms including single and multiple block write.
b)When multiple block write is stopped by CMD12, the busy from the response of CMD12 is up to 500ms.
c)When multiple block write is stopped by CMD23, the busy after the last data block is up to 500ms.
If you want to go faster use something like this Seeed Arch Max (https://developer.mbed.org/platforms/Seeed-Arch-Max/).  I have logged at over a million samples per second with its internal ADC.  I use DMA for the ADC. It has 192 KB of SRAM and a SDIO interface available.

I also like this Olimex card (https://www.olimex.com/Products/ARM/ST/STM32-E407/open-source-hardware).  It has the on board SD connected to SDIO.
Title: Re: Try this super fast analog pin logger
Post by: snoop911 on Aug 09, 2015, 01:55 am
Thanks, that Olimex card (STM32-E407) looks awesome!   I placed an order for it  ( I had been thinking of getting the F4 discovery board from aimagin, but this looks more complete)

Quote
The speed rating is for fast clock in SDIO mode, not SPI mode.
I see, so maybe only 7MBs, since SDIO is just SPI with 3 more MOSI lines?

A bit off topic, but what are your thoughts on using wireless bluetooth/wifi shields to transmit to a remote logger (pc, plotly, etc)?    I know BLE is slow (250kbps), but can't bt classic 2.1 (EDR) be used for transmitting @2Mbps using the FTP Profile?   I've seen people use bt classic with the Serial Port Profile (only 115.2kbps) but haven't found anyone using the faster classic profiles...

Or is wifi generally used for real-time wireless (1Mbps+)?  Would it be implemented as an rtp/udp server on the embedded side? 

I like the idea of offloading the data recording, but how would you transmit/receive it (wirelessly)?





Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Aug 09, 2015, 02:21 am
Quote
I see, so maybe only 7MBs, since SDIO is just SPI with 3 more MOSI lines?
No, SDIO is a totally different protocol.

UHS-I cards can have a 208 MHz clock and transfer 104 MB/sec.

SPI is limited to 50MHz clock but AVR has a max transfer rate of about 600 KB/sec.

With Due you can get about 4 MB/sec but only if you do 32 KB multi-block transfers with DMA.  You will never come close with SD.h or SdFat with the standard API.

Modern SD cards emulate 512 byte blocks and have huge overhead in SPI mode for single block transfers.

I don't know about a high speed wireless for Arduino.

I can get 95% of the bandwidth of 100 mbit Ethernet with the STM32 boards above.  They have great Ethernet controllers and the LwIP stack has optimized drivers.  See This (http://forum.chibios.org/phpbb/viewtopic.php?f=3&t=2366).
Title: Re: Try this super fast analog pin logger
Post by: Elliott12 on Aug 13, 2015, 01:47 pm

I wish I had the Sd Fat running on the 8MB Ram disk
Results are amazing
Thanks all
Title: Re: Try this super fast analog pin logger
Post by: rolfmeesters on Aug 15, 2015, 03:06 pm
i'm using the 'old' Sdfat in combination with nilRtos, in a sketch which does not log but reads a datafile to feed motor's and pins.
it took me quite some time to get it running (i'm not well versed in C).

my question: should i change to this new Sdfat lib, or is it ok to stay with the 'old' one?

reason for the question: other artists  are involved in this project, who are depending on me for the programming and are very happy with something that works sufficiently, no matter how old or up-to-date.

we use  Uno's with Adafruit Sd-shields.

thx in advance,
rolf

Title: Re: Try this super fast analog pin logger
Post by: junhanece on Sep 30, 2015, 12:17 am
Thanks so much for such a useful tool. I have successfully logged my sensor data at 40 KHz on the SD card. I have one issue though, when logging at a high frequency (e.g., 40 KHz). I get regular noise that I cannot seem to eliminate. At first, I thought it was somehow related to LED blinking, but I am not so sure if it is. Did anyone find this problem as well? Please let me know if you found a solution to this, or any guesses on what the cause may be. Thanks!
Title: Re: Try this super fast analog pin logger
Post by: sabirmgd on Feb 03, 2016, 10:07 pm
will this library work if I used an SD Card Breakout Board?
http://www.cytron.com.my/p-bb-sd-0071

thanks
Title: Re: Try this super fast analog pin logger
Post by: Gdunge on Feb 09, 2016, 11:25 pm
my question: should i change to this new Sdfat lib, or is it ok to stay with the 'old' one?
If SD is working for you, then why change? SD works for many people. It's easy to use, and already installed in the Arduino software.

If you want more functionality or more speed than SD can provide, then look at using SdFat.

Personally, I switched to SdFat to get more speed AND to get the ability to rename files. However, I found a few issues that I had to solve before my old SD-dependent code would run with SdFat. It's not necessarily painless to switch (although it's pretty straightforward if you're not using other libraries like TMRpcm or Streams that work differently or not at all with SdFat).

On the plus side, the author of SdFat is active on this forum and gave me help, along with many other people who know the Arduino way better than I do.

You need to answer the question for yourself if SD is "OK" for you.

IMO, of course.
Title: Re: Try this super fast analog pin logger
Post by: sqirry on Mar 09, 2016, 12:31 pm
Hello Fat16Lib,

you made a outstanding peace of software squeezing unbelievable performance out of the UNO.
Thanks a lot for sharing.

I intend to use your fantastic work for a portable current logger to measure loads of devices in vehicles. Tha aim is to reduce wire sizes and save copper weight by using intelligent methods to define fuse strategies for dynamic electric loads. Hence do something for reducing the CO² footprint.

Your data logger worked out of the box with my Uno and a Micro-SD Adapter that came with a cheap 2.8 inch touch TFT from China. Will make a separate Adapter now since the TFT is blocking the analog inputs.

Great work.

sqirry


Title: Re: Try this super fast analog pin logger
Post by: arihant122 on Apr 30, 2016, 10:13 am
Is there a way to dump the serial data directly to spreadsheet file like MS Excel without any compromise on speed of data logging?
That would make the life much easier.

Thanks a ton for writing this piece of code anyway.

Title: Re: Try this super fast analog pin logger
Post by: ARDPrash on Jul 24, 2016, 03:11 pm
I have used this code with Arduino Due. But it won't compile and returns an error message saying "error too little SRAM"
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Jul 24, 2016, 04:00 pm
AnalogBinLogger is an AVR only program.   Here is what you should get on Due

Quote
Arduino: 1.6.9 (Windows 10), Board: "Arduino Due (Programming Port)"

AnalogBinLogger:830: error: #error This program is only for AVR.

 #error This program is only for AVR.

  ^

exit status 1
#error This program is only for AVR.

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.
AnalogBinLogger depends on the hardware features of the AVR ADC.

The LowLatencyLogger example will work with Due and can log at 5,000 samples per second with four adcs in each sample.
Title: Re: Try this super fast analog pin logger
Post by: hugsy on Jul 31, 2016, 05:09 pm
I'm trying to use the AnalogBinLogger example from the SdFat library but it doesn't work on my Adafruit Feather 32u4 Adalogger, I can compile and upload it but there is no serial print.... The example Lowlatencylogger works fine by the way....

I have another question, does AnalogBinLogger work with SPI input from ADXL345, is there any example anywhere ?

Thank you
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Jul 31, 2016, 08:51 pm
The problem may be the USB in the 32u4.

The AnalogBinLogger only works with the AVR ADC.  It depends on the ADC digitizing while the CPU processes SD activity.  The ADC is read quickly in an ISR.

It's sad that DIY boards with the Arduino IDE are so limited.  I log data on STM32 development boards with the ChibiOS RTOS at over a million samples per second.  Boards like the $13 Nucleo STM32F411 have multiple SPI buses and high power DMA controllers.  You can also access the SD on a 4-bit wide 48 MHz SDIO bus.  

ChibiOS's excellent HAL, multi-threading and priority based preemptive scheduler make it possible to implement high performance embedded systems.

Unfortunately the skills required to use these boards properly with ChibiOS are beyond those of most DIY users.

I continue to support SdFat on Arduino but rarely use Arduino for my projects.

Title: Re: Try this super fast analog pin logger
Post by: hugsy on Aug 01, 2016, 02:26 pm
Ok thanks, I will switch to analog accelerometer... I'm gonna try again, it should work on the 32u4, at least the interface because it's quite the same code as the Lowlatencylogger.


Edit: It's working. The following command was missing:

 
Code: [Select]
while(!Serial.available()) {
    SysCall::yield();
  }


I managed to log at 17kH on one channel this code is a beauty !!
Title: Re: Try this super fast analog pin logger
Post by: ARDPrash on Aug 08, 2016, 07:22 am
I am trying to use Analog pin logger to log data with a standalone Arduino, without using a PC. Can anyone help to modify the code such that, it can log the data and convert it to .CSV format for a time period specified by me, for example, one day.
Title: Re: Try this super fast analog pin logger
Post by: fat16lib on Aug 19, 2016, 09:58 pm
I posted a new LowLatencyLogger in SdFat-beta (https://github.com/greiman/SdFat-beta).

It has several new commands.
Quote
FreeStack: 1290
Records/block: 42

type:
b - open existing bin file
c - convert file to csv
d - dump data to Serial
e - overrun error details
l - list files
r - record data
t - test without logging
the 'l' command gives a directory listing with the size and file names.
Quote
ls:
     13312 adc4pin00.bin
     11264 adc4pin01.bin
     45568 adc4pin02.bin

The 'b' command allows an existing file to be opened by entering the two digit file version.  You can then dump the file to Serial or convert it to csv.

Quote
Enter two digit version
adc4pin00.bin
File opened
Enter 'c' to convert the file to csv.
Quote
FreeStack: 706
Writing: adc4pin00.csv - type any character to stop
92%
Done: 1.20 Seconds
Here is a list of files using the 'l' command that shows the csv file.
Quote
ls:
     22624 adc4pin00.csv
     13312 adc4pin00.bin
     11264 adc4pin01.bin
     45568 adc4pin02.bin
Finally the 't' command just acquires data records at five samples per second and prints them to serial until a character is entered.

Quote
Testing - type any character to stop

micros,adc0,adc1,adc2,adc3
0,745,604,516,512
200000,571,549,510,507
400000,533,522,498,498
600000,515,504,486,488

Title: Re: Try this super fast analog pin logger
Post by: hugsy on Aug 27, 2016, 05:50 pm
I'd like to start and stop the data logging with a switch button (in addition to the serial mode). I managed to stop the logging thanks to one of your previous answer but I don't know where to put the command to start the acquisition...

I tried this but without success...


Code: [Select]
void loop(void) {
 // Read any Serial data.
if (digitalRead(switchPin) == LOW) {
   logData();   
   }
 do {
   delay(10);
 } while (Serial.available() && Serial.read() >= 0);
 Serial.println();
 Serial.println(F("type:"));
 Serial.println(F("c - convert file to csv"));
 Serial.println(F("d - dump data to Serial"));
 Serial.println(F("e - overrun error details"));
 Serial.println(F("r - record ADC data"));


I have also another issue: it doesn't work when I use the A0, A1 etc... pins with the PIN_LIST function...

Thanks for your help @fat16lib
Title: Re: Try this super fast analog pin logger
Post by: sijo on Oct 24, 2016, 05:34 pm


After executing bintocsv.exe in my PC , im getting an error "Invalid meta data".
Im using windows 10 64 bit OS. Do i need to change any setting in my  PC?

Title: Re: Try this super fast analog pin logger
Post by: ARDPrash on May 15, 2017, 09:42 am
What are the factors that affects the upper limit of the sampling rate we are using to log the data. Which code segments are responsible for the sample collection procedure that affects the sampling rate?
Title: Re: Try this super fast analog pin logger
Post by: ARDPrash on May 23, 2017, 03:44 pm
I am using this code with Arduino Mega to log data. Specifications are Arduino Mega2560, SD card size 8GB, type SDHC.

When I am try to increase my Sampling rate beyond 5 kHz, it states that the sampling rate is too high. But, I read through this forum and found that we can go up to 40 kHz for a single pin.

Is there any other factors that affecting the maximum sampling rate other than clock rate?
Title: Re: Try this super fast analog pin logger
Post by: kellyett on Aug 14, 2017, 04:57 am
I'm realy keen on being able to convert the bin  file to decimal on a PC but am struggling. I don't have much experience with bin files and am relatively new to C.

Is there an easy way in something like MATLab or VBA that I can read the bin and convert it to a dececimal?
Title: Re: Try this super fast analog pin logger
Post by: ARDPrash on Aug 14, 2017, 11:56 am
I am using this code with Arduino Mega to log data. Specifications are Arduino Mega2560, SD card size 8GB, type SDHC. The sampling frequency is 5000 Hz.

I tried to log some data continuously for one week time. But, the logging process couldn't completed as intended and it left some data for only about two days in "tmp_log.bin" file.

What is the maximum duration of datalogging for the above specified details?
Title: Re: Try this super fast analog pin logger
Post by: HWGuy007 on Apr 22, 2018, 08:09 pm
The problem may be the USB in the 32u4.

The AnalogBinLogger only works with the AVR ADC.  It depends on the ADC digitizing while the CPU processes SD activity.  The ADC is read quickly in an ISR.

It's sad that DIY boards with the Arduino IDE are so limited.  I log data on STM32 development boards with the ChibiOS RTOS at over a million samples per second.  Boards like the $13 Nucleo STM32F411 have multiple SPI buses and high power DMA controllers.  You can also access the SD on a 4-bit wide 48 MHz SDIO bus.  

ChibiOS's excellent HAL, multi-threading and priority based preemptive scheduler make it possible to implement high performance embedded systems.

Unfortunately the skills required to use these boards properly with ChibiOS are beyond those of most DIY users.

I continue to support SdFat on Arduino but rarely use Arduino for my projects.


Thanks for your contributions, fat16lib.  Your work is great!  

I have been studying your AnalogBinLogger project for the Mega 2560 for a few days.  I have been using this project in order to collect analog data.  I modified the code so that I only log data based on a digital input pin that I use as a Gate.  This allows me to not store samples when the gate pin is low, which in my case is 2 seconds long.  I wanted to sample the analog inputs as fast as possible.

I then discovered that the maximum sampling rate I could accomplish with the 2560 is one pin every 25 usec.  This worked well.  However, I wanted to take the example to the next level and possibly sample a single pin at 1 sample every 1 us (of course this figure would get diluted as I add more pins).

In order to sample this fast, I focused on the DUE, which can sample at 1 Msps.  So I started bending the code to prepare it for the DUE.  However, today I discovered your post where you praise the $13 STM32 board and the ChibiOS environment.

I have a few questions, please:

1. Using the $13 STM32 board you mention, will I be able to use ChibiOS and the development environment to compile, link and more importantly, debug my code?  I have used RTOSes in the past (ThreadX, RTXC), so I am a little with RTOS concepts.

2. Have you written an open-source example for the AnalogBinLogger for this board?  That would be great.

3. For logging Analog data onto an SD card at say 1 pin at 1 MSPS, which of the two boards do think can do it faster and is easier to develop with - the DUE or $13 STM32 board?

Regards.
Title: Re: Try this super fast analog pin logger
Post by: ard_newbie on Apr 24, 2018, 06:33 am
3. For logging Analog data onto an SD card at say 1 pin at 1 MSPS, which of the two boards do think can do it faster and is easier to develop with - the DUE or $13 STM32 board?

Regards.
For the DUE (~ $15), with the HSMCI interface, the maximum performance seems to be (SD clock 48 MHz):

Write 18.75MBytes/sec.
Read 19.5MBytes/sec.


https://forum.arduino.cc/index.php?topic=188360.0 (https://forum.arduino.cc/index.php?topic=188360.0)
Title: Re: Try this super fast analog pin logger
Post by: Ross2511 on Oct 08, 2018, 12:54 pm
Hello,

Thanks for the awesome code, i'm successfully recording at 8 kHz using the uno and the adafruit data shield.

My measurement system is going to used to detect coughs and ideally id like to call the time stamp about once every minute or five minutes just to confirm the sampling frequency.

Is it possible to write the time stamp in binary? Would it be easier to utilise the seconds from midnight 01/01/1970 function that the adafruit data logger has?

Thanks

Ross
Title: Re: Try this super fast analog pin logger
Post by: Ross2511 on Nov 27, 2018, 09:57 am
Following on from my previous post i have been running experiments to evaluate if the sampling intervals are stable enough to allow me to only use the sample number as the time. My project involves an electret microphone recording audio for cough detection.

Using a digital clock with milliseconds i would sound a buzzer directly above the microphone (for 1s) to register a 5V signal. I then used MATLAB to identify the time of these peaks using the sampling interval along with the sample number and check them against the time i've recorded applying the beep.

Although no overruns are reported when the recording ends the times of the beeps identified on MATLABb seem to almost drop by 4s an hour. I am running these tests for between 3-4 hours but eventually would like to run this project for 24 hours.

I initially was recording at 10kHz with the clock running at 250kHz. I then tried to increase clock speed to 500kHz but that seemed to increase the error.

Finally i decreased to 5kHz sampling frequency to 5kHz with a clock speed of 125kHz.

Has anyone else had experience with this?

Thanks in advance for any suggestions, i have attached screenshots of the result tables.