Show Posts
Pages: 1 2 3 [4] 5 6 ... 99
46  Using Arduino / Sensors / Logging analog pins at 40,000 samples per second. on: March 26, 2014, 07:11:56 am
I have posted a high speed analog pin to SD logger here http://forum.arduino.cc/index.php?topic=228549.0.

I posted it under storage since most of the code is related to the SD.

I tested it by logging five pin samples at 5,000 samples per second or 25,000 ADC values per second.

I also did single pin tests at 40,000 samples per second.

Please post post any comments at the above location.

moderator: locked to topic to prevent reactions in this thread
47  Using Arduino / Storage / Re: Try this super fast analog pin logger on: March 26, 2014, 07:01:49 am
I have updated the zip file attached to the first post to AnalogBinLogger20140326beta.zip.

I now determine the amount of buffering by use of the RAMEND symbol.  This allows the logger to run on ATmega chips with 2 KB or more of SRAM.

I also fixed a problem that often causes an overrun due to SD write latency to be fatal.

I would appreciate any comments on results with various SD card.  I have had the best luck with SanDisk class 4 cards.

This is the card I used in development http://www.amazon.com/dp/B007JRB0RY.  I tested five of these cards, two 4 GB, two 8 GB, and one 32 GB.  I have had almost no overrun errors on an Uno with these cards.

I have done some testing with this microSD http://www.amazon.com/Sandisk-MicroSDHC-Memory-Card-Adapter/dp/B000WH6H1M It appears to work but I have no done a long reliability test.

Here is the result of a short 100 MB session with the 8 GB class 4 SanDisk  micro SD where the maximum write latency was 920 usec for a 512 byte block.
Quote
Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG01.BIN
Max block write usec: 920
Record time sec: 1993.845
Sample count: 9969226
Samples/sec: 5000.00
Overruns: 0

Here is one of the worst cards http://www.amazon.com/PNY-Optima-Class-Memory-P-SDHC4G4-EF/dp/B000L7INLU.  I use it when I want to be sure of overruns.
48  Using Arduino / Storage / Re: Try this super fast analog pin logger on: March 25, 2014, 04:06:02 pm
I tested on an Uno and a Mega.  The program is designed to work with other AVR boards.

To use other boards a section needs to add to this area to define BUFFER_BLOCK_COUNT and QUEUE_DIM.
Code:
#if defined(__AVR_ATmega328P__)
// 328 cpu -  use total of two 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 1;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 4;  // Must be a power of two!
//
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
// Mega - use total of 13 512 byte buffers
const uint8_t BUFFER_BLOCK_COUNT = 12;
// Dimension for queues of 512 byte SD blocks
const uint8_t QUEUE_DIM = 16;  // Must be a power of two!
//
#else  // CPU types
#error Undefined CPU
#endif  // CPU types

I plan to use the RAMEND symbol to define these symbols in the next version.

I tested with a number of SD shields/modules.  
You just need to edit this for your shield/module.
Code:
// SD chip select pin.
const uint8_t SD_CS_PIN = SS;

On Uno I mostly used an Adafruit data-logging shield.  It has two LEDs and I used one for an error indicator.
Code:
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;
49  Using Arduino / Storage / Try this super fast analog pin logger on: March 25, 2014, 03:00:54 pm
I have attached a beta version of a very fast data logger for analog pins.  I would appreciate any feedback from those who try this program.

The tests described below were done on an Uno.

Here is part of the readme file.
Quote
AnalogBinLogger.ino logs analog data to a binary SD file at high rates.

Samples are logged at regular intervals by using timer1.  Timer/Counter1
Compare Match B is used to trigger the ADC for the first pin in a sample.
The ADC is triggered for remaining sample pins in the ADC conversion complete
interrupt routine.

Data is captured in the ADC interrupt routine and saved in 512 byte buffers.

Buffered data is written to the SD in a function called from loop().  The
entire data set is written to a large contiguous file as a single multi-block
write.  This reduces write latency problems.

Many inexpensive SD cards work well at lower rates.  I used a $6.00
SanDisk 4 GB class 4 card for testing.

SanDisk class 4 cards work well at fairly high rates.  I used the 4 GB SanDisk
card to log a single pin at 40,000 samples per second.

The bintocsv folder contains a PC program for converting binary files to
CSV files.  I have included a executable for Windows.  Linux and Mac users
can build from the included source files.  bintocvs is a command line program.

bintocsv binFile csvFile

The attached file, DATA.png, is a plot of a 2 kHz sine wave logged at 40,000 samples per second.  FFT.png shows a FFT of this data.

I did a reliability test logging five analog pins at 5,000 samples per second.  This is an ADC rate of 25,000 values per second.  I logged 512 MB of data without dropping any values.

Serial output from the test.
Quote
Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG10.BIN
Max block write usec: 920
Record time sec: 10239.992
Sample count: 51199950
Samples/sec: 5000.00
Overruns: 0
Done

Here is the first part of the csv file produced from the test by the included bintocvs  program.
Quote
Interval,200.0000,usec
pin0,pin1,pin2,pin3,pin4
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
0,1023,0,670,0
Pins 0, 2, 4 are connected to ground, pin 1 to 5V and pin 3 to 3.3V.

Here is the configuration section of the logger.
Code:
//------------------------------------------------------------------------------
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers can be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};
//------------------------------------------------------------------------------
// Sample rate in samples per second.
const float SAMPLE_RATE = 5000;  // Must be 0.25 or greater.

// The interval between samples in seconds, SAMPLE_INTERVAL, can be set to a
// constant instead of being calculated from SAMPLE_RATE.  SAMPLE_RATE is not
// used in the code below.  For example, setting SAMPLE_INTERVAL = 2.0e-4
// will result in a 200 microsecond sample interval.
const float SAMPLE_INTERVAL = 1.0/SAMPLE_RATE;

// Setting ROUND_SAMPLE_INTERVAL non-zero will cause the sample interval to
// be rounded to a a multiple of the ADC clock period and will reduce sample
// time jitter.
#define ROUND_SAMPLE_INTERVAL 1
//------------------------------------------------------------------------------
// ADC clock rate.
// The ADC clock rate is normally calculated from the pin count and sample
// interval.  The calculation attempts to use the lowest possible ADC clock
// rate.
//
// You can select an ADC clock rate by defining the symbol ADC_PRESCALER to
// one of these values.  You must choose an appropriate ADC clock rate for
// your sample interval.
// #define ADC_PRESCALER 7 // F_CPU/128 125 kHz on an Uno
// #define ADC_PRESCALER 6 // F_CPU/64  250 kHz on an Uno
// #define ADC_PRESCALER 5 // F_CPU/32  500 kHz on an Uno
// #define ADC_PRESCALER 4 // F_CPU/16 1000 kHz on an Uno
// #define ADC_PRESCALER 3 // F_CPU/8  2000 kHz on an Uno (8-bit mode only)
//------------------------------------------------------------------------------
// Reference voltage.  See the processor data-sheet for reference details.
// uint8_t const ADC_REF = 0; // External Reference AREF pin.
uint8_t const ADC_REF = (1 << REFS0);  // Vcc Reference.
// uint8_t const ADC_REF = (1 << REFS1);  // Internal 1.1 (only 644 1284P Mega)
// uint8_t const ADC_REF = (1 << REFS1) | (1 << REFS0);  // Internal 1.1 or 2.56
//------------------------------------------------------------------------------
// File definitions.
//
// Maximum file size in blocks.
// The program creates a contiguous file with FILE_BLOCK_COUNT 512 byte blocks.
// This file is flash erased using special SD commands.  The file will be
// truncated if logging is stopped early.
const uint32_t FILE_BLOCK_COUNT = 256000;

// log file base name.  Must be six characters or less.
#define FILE_BASE_NAME "ANALOG"

// Set RECORD_EIGHT_BITS non-zero to only record the high 8-bits of the ADC.
#define RECORD_EIGHT_BITS 0
//------------------------------------------------------------------------------
// Pin definitions.
//
// Digital pin to indicate an error, set to -1 if not used.
// The led blinks for fatal errors. The led goes on solid for SD write
// overrun errors and logging continues.
const int8_t ERROR_LED_PIN = 3;

// SD chip select pin.
const uint8_t SD_CS_PIN = SS;
50  Using Arduino / Storage / Don't Format SD cards with OS utilities! on: March 24, 2014, 07:53:11 am
Always format SD cards using SD Formatter https://www.sdcard.org/downloads/formatter_4/.

NEVER format SD card with OS utilities!

Here are some examples that show what can happen.  First is an extreme case for determining free space on an SD.  This type performance hit will also be true for seek.

Time to determine free space with a properly formatted card using the SdInfo sketch.  See http://forum.arduino.cc/index.php?topic=228128.msg1648437#msg1648437 for more details on code.

Time 0.302 seconds.
Quote
millis: 302
freeClusters: 63883
freeSpace: 2093.32 MB (MB = 1,000,000 bytes)

Same card formatted FAT32 with small clusters on Windows.
34.885 seconds.
Quote
millis: 34885
freeClusters: 4033302
freeSpace: 2065.05 MB (MB = 1,000,000 bytes)

Here are results running the SdFat bench sketch on this card.  Write takes about four times longer with FAT32 and small clusters.

Properly formatted card.
Quote
Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 358.47 KB/sec
Maximum latency: 58888 usec, Minimum Latency: 84 usec, Avg Latency: 273 usec

Starting read test.  Please wait up to a minute
Read 369.19 KB/sec
Maximum latency: 1912 usec, Minimum Latency: 84 usec, Avg Latency: 265 usec

Same card formatted FAT32 with small clusters on Windows.
Quote
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 89.43 KB/sec
Maximum latency: 117636 usec, Minimum Latency: 84 usec, Avg Latency: 1113 usec

Starting read test.  Please wait up to a minute
Read 225.44 KB/sec
Maximum latency: 1920 usec, Minimum Latency: 80 usec, Avg Latency: 437 usec

These are extreme cases but I do often see this type problem so please use this formatter https://www.sdcard.org/downloads/formatter_4/.  It will always produce the optimum format for your card.
51  Using Arduino / Storage / Re: Free space counter is slow on: March 24, 2014, 06:46:20 am
Quote
I timed it 10 seconds before the free space was displayed.

I added timing to the SdFat SdInfo sketch and tried several cards.  Here is the SdInfo code:
Code:
 uint32_t m = millis();
  uint32_t volFree = vol.freeClusterCount();
  m = millis() - m;
  cout << pstr("millis: ") << m << endl;
  cout << pstr("freeClusters: ") <<  volFree << endl;
  float fs = 0.000512*volFree*vol.blocksPerCluster();
  cout << pstr("freeSpace: ") << fs << pstr(" MB (MB = 1,000,000 bytes)\n");

Here are results with SdInfo's 4 MHz SPI clock:

1.9 seconds for a 4 GB SanDisk class 4 card.
Quote
millis: 1900
freeClusters: 97017
freeSpace: 3179.05 MB (MB = 1,000,000 bytes)

0.487 seconds for a 2 GB SanDisk class 2 card.
Quote
millis: 487
freeClusters: 59359
freeSpace: 1945.08 MB (MB = 1,000,000 bytes)

0.438 seconds for a 2 GB Dane-Elec Card.
Quote
millis: 438
freeClusters: 61357
freeSpace: 2010.55 MB (MB = 1,000,000 bytes)

I ran the test again with a 8 MHz SPI clock.

0.362 seconds for the 2 GB SanDisk card.
Quote
millis: 362
freeClusters: 59359
freeSpace: 1945.08 MB (MB = 1,000,000 bytes)

1.403 seconds for the 4 GB SanDisk card.
Quote
millis: 1403
freeClusters: 97017
freeSpace: 3179.05 MB (MB = 1,000,000 bytes)

These tests show an interesting fact.  Access to FAT16 on the 2 GB cards is faster since the FAT entry size is 16-bits.  4 GB and larger cards have 32-bit FAT entries so twice as much I/O is required.  That's why the 4 GB card takes about four times as long as the 2 GB cards.

Often people make the mistake of formatting small card FAT32 and ruin performance.

So always use SD formatter https://www.sdcard.org/downloads/formatter_4/.  It will always produce the best results.

NEVER use OS utilities with your idea of what is best for SD card performance.

Here is the 2 GB Dane-Elec card formatted FAT32 with 1 KB clusters on Windows.  The SPI speed is 8 MHz.

18.228 seconds.
Quote
millis: 18228
freeClusters: 1949567
freeSpace: 1996.36 MB (MB = 1,000,000 bytes)
fatStartBlock: 2559
52  Using Arduino / Storage / Re: Stack address > 2k for ATMEGA328p? on: March 21, 2014, 05:51:44 pm
Looks like mpflaga https://github.com/mpflaga/Arduino-MemoryFree/blob/master/MemoryFree.cpp lifted it from an old version of SdFat.

I fixed it about a year ago.

Here is the real freeMemory http://playground.arduino.cc/Code/AvailableMemory#.UyzEm_BX-uY

freeMemory() in this library gives stack and heap memory.
53  Using Arduino / Storage / Re: Stack address > 2k for ATMEGA328p? on: March 21, 2014, 05:32:07 pm
Quote
Now i really wish i had kept a copy of the original code

Give it up.  I wrote that code.  It was first posted in SdFat in May 2013.

If you did download it where is it located?  I would like to know who first lifted it.  Did they remove the gnu licence?

My code:
Code:
#ifdef __arm__
// should use uinstd.h to define sbrk but Due causes a conflict
extern "C" char* sbrk(int incr);
#else  // __ARM__
extern char *__brkval;
extern char __bss_end;
#endif  // __arm__
//------------------------------------------------------------------------------
/** Amount of free RAM
 * \return The number of free bytes.
 */
int SdFatUtil::FreeRam() {
  char top;
#ifdef __arm__
  return &top - reinterpret_cast<char*>(sbrk(0));
#else  // __arm__
  return __brkval ? &top - __brkval : &top - &__bss_end;
#endif  // __arm__
}

Your code:
Code:
#ifdef __arm__
// should use uinstd.h to define sbrk but Due causes a conflict
extern "C" char* sbrk(int incr);
#else  // __ARM__
extern char *__brkval;
extern char __heap_start;
#endif  // __arm__

int freeMemory() {
  char top;
#ifdef __arm__
  return &top - reinterpret_cast<char*>(sbrk(0));
#elif defined(CORE_TEENSY)
  return &top - __brkval;
#else  // __arm__
  return __brkval ? &top - __brkval : &top - &__heap_start;
#endif  // __arm__
}

54  Using Arduino / Storage / Re: Stack address > 2k for ATMEGA328p? on: March 21, 2014, 05:09:20 pm
Quote
All I did was take the a MemoryFree library that I downloaded off the Arduino website and edit it until it worked for me.

You lifted the SdFat code.  You would never have put this in.
Code:
// should use uinstd.h to define sbrk but Due causes a conflict

Why are you claiming you wrote your "fix"?
55  Using Arduino / Storage / Re: Using sdfatlib with getline and seekg? on: March 21, 2014, 04:16:56 pm
Post your code.
56  Using Arduino / Storage / Re: Stack address > 2k for ATMEGA328p? on: March 21, 2014, 03:54:27 pm
Your "fix" looks a lot like my SdFat code from 5/4/2013.  What do you claim is new?
Quote
04 May 2013

Fix FreeRam() for 1.05/1.53 malloc.

Code:
/* Arduino SdFat Library
 * Copyright (C) 2012 by William Greiman
 *
 * This file is part of the Arduino SdFat Library
 *
 * This Library is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * This Library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.

 * You should have received a copy of the GNU General Public License
 * along with the Arduino SdFat Library.  If not, see
 * <http://www.gnu.org/licenses/>.
 */
#include <stdlib.h>
#include <SdFat.h>
#include <SdFatUtil.h>
#ifdef __arm__
// should use uinstd.h to define sbrk but Due causes a conflict
extern "C" char* sbrk(int incr);
#else  // __ARM__
extern char *__brkval;
extern char __bss_end;
#endif  // __arm__
//------------------------------------------------------------------------------
/** Amount of free RAM
 * \return The number of free bytes.
 */
int SdFatUtil::FreeRam() {
  char top;
#ifdef __arm__
  return &top - reinterpret_cast<char*>(sbrk(0));
#else  // __arm__
  return __brkval ? &top - __brkval : &top - &__bss_end;
#endif  // __arm__
}

freeMemory returns free memory for both the heap and stack.  My (not likely your) FreeRam() only returns free memory for the stack.
57  Using Arduino / Storage / Re: Sdfatlib error on Arduino Due on: March 20, 2014, 12:12:49 pm
The partition table is reading as junk.  It may be a wiring problem since there is no hardware CRC on transfers to detect bad SPI signals.

Quote
I double checked the SPI Slave Select pins and tried to lower the SPI clock, but still no luck.

How much did you lower the SPI clock?

SdFat uses very fast DMA SPI.  It is about 10 times faster than the old SdFat included in the default SD.h.  This means you need very clean short wires for the new SdFat.

Here is the difference in a benchmark.

SD.h
Quote
File size 5MB
Buffer size 8192 bytes
Starting write test.  Please wait up to a minute
Write 240.00 KB/sec
Maximum latency: 54003 usec, Minimum Latency: 32106 usec, Avg Latency: 34119 usec

Starting read test.  Please wait up to a minute
Read 444.27 KB/sec
Maximum latency: 19585 usec, Minimum Latency: 18356 usec, Avg Latency: 18436 usec

The new SdFat:
Quote
File size 5MB
Buffer size 16384 bytes
Starting write test.  Please wait up to a minute
Write 3823.35 KB/sec
Maximum latency: 19636 usec, Minimum Latency: 4105 usec, Avg Latency: 4311 usec

Starting read test.  Please wait up to a minute
Read 4150.43 KB/sec
Maximum latency: 6475 usec, Minimum Latency: 3915 usec, Avg Latency: 3944 usec

You can enable software CRC to check for SPI errors.

Edit SdFatConfig.h and change line 63 to this:
Code:
#define USE_SD_CRC 2

This will check for SPI I/O errors but slow transfers to something like this:
Quote
File size 5MB
Buffer size 16384 bytes
Starting write test.  Please wait up to a minute
Write 1601.64 KB/sec
Maximum latency: 16326 usec, Minimum Latency: 10017 usec, Avg Latency: 10186 usec

Starting read test.  Please wait up to a minute
Read 1670.72 KB/sec
Maximum latency: 12332 usec, Minimum Latency: 9775 usec, Avg Latency: 9806 usec
58  Using Arduino / Storage / Re: sdfatlib: Possible to work around or minimize high-latency reads? on: March 20, 2014, 07:34:06 am
I just looked at some tests I did for a similar project.  This project needed to read data from an SD and send data to a device over SPI at very high rates.

My solution was to read a contiguous file using the raw SD interface.  The standard SPI controller is dedicated to the SD.

With an industrial SD I could read the file at about 600 KB/sec with a maximum read latency of 1032 micros for a 512 byte block.
Quote
Starting read test
Read Done
File size: 51200 KB
Elapsed time: 81.950 seconds
Max read time: 1032 micros
Read rate: 624.773 KB/sec

I used an USART in SPI mode to send data to the device.  An AVR USART can run at 8 MHz in SPI mode.

I used a small RTOS, NilRTOS https://code.google.com/p/rtoslibs/, that I ported to Arduino.  This allows the thread that sends data to the device to run at high priority and the SD read thread to fill buffers at lower priority.

I did tests assuming a Mega or 1284P board would be used.  I am sure you could send 150 KB/sec with a Mega.   It might be possible to achieve this rate with an Uno but it would be marginal due to the limited 2 KB of SRAM.

Lower cost SD card can have occasional longer read latencies but would work with a Mega or 1284.

Here is a read result for a $5.00 4 GB SanDisk class 4 card.
Quote
Read Done
File size: 51200 KB
Elapsed time: 81.252 seconds
Max read time: 2488 micros
Read rate: 630.135 KB/sec
The 2488 micros would not be a problem with more buffering.
59  Using Arduino / Storage / Re: sdfatlib: Possible to work around or minimize high-latency reads? on: March 19, 2014, 10:07:13 pm
Does your display interface use the SPI bus?

You can't use SPI in an ISR while a program like SdFat is doing a transfer in non-interrupt code.
60  Using Arduino / Storage / Re: sdfatlib: Possible to work around or minimize high-latency reads? on: March 19, 2014, 03:33:38 pm
You could use the same kind of buffering that audio players use.

I wrote a wave file player about five years ago that plays 16-bit 44.1 ksps wave files http://code.google.com/p/wavehc/downloads/list.  This player delivers data to an DAC at a constant rate of 88.2 KB/sec using a 328 Arduino.

This player uses a timer1 IRS to send data from buffers to the DAC.  Data is read from the SD into buffers in non-interrupt code.

Define your needs as pito suggested.

What is the data rate and how is the data sent to your POV display?  Give details of the interface to the display.
Pages: 1 2 3 [4] 5 6 ... 99