Show Posts
Pages: 1 ... 61 62 [63] 64 65 ... 109
931  Using Arduino / Storage / Re: Unformatted write to SD on: August 19, 2012, 01:26:25 pm
What appears sub-optimal?  I just stuck the lines in the loop to test sleep, not as a test of jitter.

I didn't post the Noise Reduction tests.

Appears your C++ is a bit rusty. The first statement is a type declaration so the library will compile.  I could have put it anywhere before the call.
  int rawAnalogReadWithSleep();
This is the call:

The "no-op" ISR is necessary to field the wake-up interrupt.

Most of the time in the loop is spent sleeping.  The person that wrote this function is allowing for wake-up by interrupts other than the ADC.  If the ADC is not done, the function goes back into ADC Noise Reduction Mode.

"no-op" ISRs are not uncommon.  Sometimes they clear a flag or cause other status change. They are very fast since no context needs to be saved.  I use one to clear a timer flag in the 100,000 sample per second logger.

I did more testing on the ADC Noise Reduction Mode.  I used a high resolution DAC to generate a ramp.  The DAC is on a well designed shield on the Arduino I was testing.

I got the noise so low with just analogRead() that I couldn't do oversampling.  For a number of DAC steps the 10-bit Arduino ADC always gives the same value.  I don't need noise reduction, I need noise injection to make oversampling work.
932  Using Arduino / Storage / Re: Unformatted write to SD on: August 19, 2012, 10:21:30 am
I did a test and the good news is reading the ADC in Noise Reduction Mode in the middle of an SD block transfer seems to work OK.

I did not write rawAnalogReadWithSleep() but it seems to use Noise Reduction Mode.

I added the code between the slashes to the write loop (two bytes are sent for each pass to slightly speed the transfer):
 for (uint16_t i = 0; i < 512; i += 2) {
    while (!(SPSR & (1 << SPIF)));
    int rawAnalogReadWithSleep();
    if (i == 300) {
    SPDR = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i + 1];
I ran this sketch:
#include <SdFat.h>
#include <avr/sleep.h>
SdFat sd;
SdFile file;

ISR(ADC_vect) { }

int rawAnalogReadWithSleep() {
  // Generate an interrupt when the conversion is finished

  // Enable Noise Reduction Sleep Mode

  // Any interrupt will wake the processor including the millis interrupt so we have to...
  // Loop until the conversion is finished
    // The following line of code is only important on the second pass.  For the first pass it has no effect.
    // Ensure interrupts are enabled before sleeping
    // Sleep (MUST be called immediately after sei)
    // Checking the conversion status has to be done with interrupts disabled to avoid a race condition
    // Disable interrupts so the while below is performed without interruption
  // Conversion finished?  If not, loop.
  while( ( (ADCSRA & (1<<ADSC)) != 0 ) );

  // No more sleeping
  // Enable interrupts

  // The Arduino core does not expect an interrupt when a conversion completes so turn interrupts off
  ADCSRA &= ~ _BV( ADIE );

  // Return the conversion result
  return( ADC );

void setup() {
    // setup ADC
  if (!sd.begin())sd.initErrorHalt();
    sd.errorHalt("opening test.txt for write failed");
  for (uint16_t i = 0; i < 50000; i++) {
    if (file.writeError) sd.errorHalt("print");
void loop() {}
Lots of dots get printed and the file has the correct content.  The file has 660 blocks.

The bad news is that I can't find proof on the web that Noise Reduction Mode helps.

I tried various tests comparing analogRead() with rawAnalogReadWithSleep() above.  If there is an improvement it is really small.  Other factors overwhelm the the difference between the two functions.  Clean power to the Arduino makes a huge difference.

One curious result is that the two functions return slightly different results.  You need to average a 1000 measurements to see the difference.

My test setup was very crude so it is not definitive but as until I see proof I won't believe Noise Reduction Mode is worth the pain.

I hope you prove Noise Reduction Mode gives more accuracy.

I did some more work and oversampling and Noise Reduction Mode are extremely frustrating.  If I work hard to reduce noise, oversampling will not work since you need noise for oversampling.  I don't trust oversampling with the 10-bit AVR ADC, it's too easy to fool yourself.

If you need more accuracy, an external ADC seem like a far better approach.

933  Development / Other Software Development / Re: Fast digital I/O and software SPI with C++ templates on: August 18, 2012, 09:25:56 pm
The library in this thread didn't work well for use in other bit-bang libraries so I have rewritten it.  One of the main reasons for fast digital I/O is bit-bang for protocols like SPI.

Please see

Also see this topic,117356.0.html.

I suspect the templates kert describes above will have the same problems.

I still include template classes for simple use in sketches but I base these classes on static functions with constant arguments.

These static functions are easier to use in other bit-bang libraries like a software SPI library which is included as an example in the new library.
934  Using Arduino / Storage / Re: New fast data logging sketches on: August 18, 2012, 04:51:58 pm
If you just want simple hardware SPI access, read section 6.1 of the datasheet
935  Using Arduino / Storage / Re: Unformatted write to SD on: August 18, 2012, 02:09:31 pm
I don't log access time for read.  I provide a function that allows all time fields of a file to be set with a call like Linux/UNIX touch.

I have never been asked to provide an update to access time for reads.

For write/create I update all fields since it's free.  The directory entry only gets updated when you call sync() or close a file.  That's when time fields get updated in the directory entry.

I think the answer to why Atmel is Arduino.  Arduino give easy software/hardware access to microprocessors.

AVR is old but fun.  It reminds me of my first microprocessor hardware projects with the 6800 and 6502 (the 6502 was used in the Apple II) around 1974.

People use sleep modes with SdFat to save battery power.  They just aren't sleeping in the middle of a block transfer.  I think sleeping between blocks in multi-block mode would work.  The SD card won't know and chip select is high.
936  Using Arduino / Storage / Re: Unformatted write to SD on: August 18, 2012, 12:16:50 pm
As an engineer "forget the datasheet" is not part of my way of working. It is not a general guide , it's the bible.
As a PhD physics researcher working on some the world's largest experiments, I take datasheets with a grain of salt.  They are a clue but you must evaluate devices in the mode you use them.  Often devices change but not the datasheets.
I get the impression you are confusing a statistical measurement , ENOB, with the spec for the accuracy on one ADC conversion. The two results are compatible and not contradictory. They are different things.
Accuracy of one conversion is a statistical measurement.  Accuracy is always a statistical measurement.

Did you read the Oskar Leuthold paper on the AVR ADC?  He is an engineer working for GEC Plessey Semiconductors.  He designs fast, 200 Megasample per second  A/D converters.

What is wrong with his testing of the AVR ADC?
I am logging physical quantities, not audio.
Audio is a physical signal.  Some of the most advanced research in high quality ADCs involves audio.
Are you able to comment on whether sleeping the CPU would break the SPI streaming or is the protocol robust enough to stand  a circa 100us hiatus?
Using ADC Noise Reduction Mode is likely to cause problems with an SD transfer.  Atmel documents suggest that Idle Mode could work O.K.  The SPI controller is stopped in ADC Noise Reduction Mode and that could cause a glitch.

That's a comment not a proven fact.  My original library allowed interruption of block transfers but I kept having problems so I removed the code.

If you need accuracy at 50 Hz why not us an external ADC?  I use external sigma-delta ADCs frequently.  I wrote a fast Software SPI library that runs at about 2MHz to access external ADCs since it is difficult/impossible to share the hardware SPI bus with the SD.  

At 50 Hz you could even use low cost I2C ADCs.  Here is a family of 18-bit delta-sigma ADCs I like a lot (maybe too slow for you). They have a 0.05% on-board voltage reference and an on-board PGA.  It could give you 14 bits at 60 sps.

This may not work for your application but there are many choices that are easy to use.  Why fight with the AVR ADC if it doesn't meet your requirement.

I mainly play with pushing the AVR ADC as a game.  For serious hobby measurements I use external ADCs/sensors.  

I am now mostly using Cortex M4 STM32 processors, not Arduino.  I run ChibiOS/RT as the OS.

Here is an example board it has three really fast 12-bit ADCs that can run in parallel.
937  Using Arduino / Storage / Re: Unformatted write to SD on: August 18, 2012, 09:56:31 am
Specifically what happens when you go beyond the "management block" size of 128MB or whatever? Do you know how/why you were able to avoid hitting a busy delay?
The busy delay has nothing to do with "management block".  The big delay happens in single block mode because the controller does not plan ahead. For the 12th time I use multi-block streaming mode.
Yes, but have you tested >5MB to see whether you are still getting no busy time? 
Yes, I designed SdFat for audio recording and other high speed logging.  I have logged for hours in real apps.

Limor Fried asked me to make a version for beginning users.  The Arduino company decided it was too complex and wrapped it with their SD.h API.  That's why you think I designed SdFat for beginning user. 

You can used just the files Sd2Card.h, Sd2Card.cpp and SdInfo.h as a library.  This sketch takes 2120 bytes of flash and will write block zero of an SD.
#include <Sd2Card.h>
Sd2Card card;
uint8_t buf[512];
void setup() {
  card.writeBlock(0, buf);
void loop() {

ADC noise reduction mode which is required to get (nominal) 10b accuracy from the Atmel chip.  This scheme seems fine for your 8bit sampling but  you have to chose between higher resolution ADC and jitter.
Forget the datasheet, it is a general guide.  Look at the AVR evaluation tests. In the papers that I pointed to, the ADC is triggered by the CPU clock and noise reduction is done using the DIDR.

The result is 7.4 ENOB for the 2MHz rate used at 100,000 samples per second.  For 10-bit sampling 33 ksps gives a ENOB of about 9.3 with a 500kHz ADC clock.   The max ENOB is 9.5 for the AVR ADC in any test.

Here is a paper on SNR due to sampling jitter

Clearly the jitter in the AVR timer compare event is less that a CPU cycle.  I said it was less than a CPU cycle since I don't know the exact number.

ardnut,  If you are over sampling, why write all the data to the SD?  What kind of signal are you recording?  How can you possibly use the AVR ADC for a fast multi-channel signal?
938  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 06:56:02 pm
I ran the following sketch to check memory use.
#include <SdFat.h>
#include <SdFatUtil.h>
SdFat sd;
SdFile file;

void setup() {
  if (!sd.begin()) return;"SIZE_TST.TXT", O_RDWR | O_CREAT | O_AT_END);
void loop() {}

The file contains the value 1369.  So total used RAM is 679 bytes.  Since the 512 byte buffer can be used for logging, total RAM for the Arduino core and other SdFat use is 167 bytes.

939  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 06:04:04 pm
Part of my aim was to remove the RAM needed by the full SDfat lib
The 512 byte block cache RAM in SdFat can be used for logging with raw writes.  I use it in my fast loggers.  There is a call that flushes the block cache and returns the address of the cache.  Very little other RAM is globally allocated.
Also to get the largest contiguous block you will need to know what else is on the fs (ie have it freshly formatted) at which point some of the interest in using an fs is lost.
It's easier to use contiguous files than a raw device.  That's why the POSIX real-time file extensions were developed for RTOSs used in embedded systems.

SdFat allows up to a 4GB contiguous file to be created.  It finds the first fit place.  If you are willing to use an SD as a raw device, you will suffer more pain than formatting the SD.
I think to get the best from any given card will require some specific information about it and adapting the writing cycle to fit.
Not likely.  Better to spend some money on an industrial SD designed for embedded systems.

Doesn't that limit you to 1kS/s unless you are willing to accept some substantial jitter?
The jitter for the 100,000 sample per second logger is a less than one CPU cycle, which is 62.5 ns.

I trigger the ADC on a timer1 compare event.  I read the completed conversion in an ISR and buffer it.

The buffers are written to SD in the background.  At least 82 data points are taken during the write of an SD block.

940  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 03:14:57 pm
Wear leveling algorithms are not always a totally black box.  Look at this ]].

Tell me how it helps if you can't access the use counts and mappings.

Since every manufacturer has different internal structures and algorithms it is even harder.
941  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 01:50:38 pm
It seems that the potential 100ms busy time is a bit of a killer for what we are both trying to do.
No, I am perfectly happy with the result I posted above.  Here it is again:
Start raw write of 5120000 bytes at
256000 bytes per second
Please wait 20 seconds
Elapsed time: 20000 millis
Max write time: 828 micros
Overruns: 0
This means I can write at up to 256 KB/sec and the time to write a block is no greater than 828 usec.  There is no busy delay so about 42% of the CPU time is required.  This program simulates a data logger by writing a block every 2,000 usec in the multi-block mode I described before.

It is very difficult to write 512 bytes from RAM to the SPI bus much faster.

For typical logging applications, most of the CPU time is used acquiring data.  Writing at 100 KB/sec requires less than 20% of the CPU so 80% is available to acquire data.

The above result is for raw writes to a large contiguous file.  The fact the SD is formatted with a file system has no effect.

If you use the same SD for logging with single block writes to a file you get this result for 100 byte writes:
Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 199.21 KB/sec
Maximum latency: 86384 usec, Minimum Latency: 84 usec, Avg Latency: 496 usec
There was at least on busy delay of 86.4 ms with this SD.  The minimum latency, 84 usec, occurs when the write is just a copy to the SdFat block buffer and no write to the SD occurs.

This is one of the best SD cards around for Arduino use and it has almost a 100 ms delay.

The rate was under 200 KB/sec and required 100% CPU.

So what do you expect to achieve? 

Why do you want to use single block writes.  The above test proves that streaming multi-block writes work with good cards.
942  Using Arduino / Storage / Re: Logging 100 ksps with the Arduino internal ADC on: August 16, 2012, 11:04:53 am
The papers are included in  There are two tests of the AVR ADC.

Have you read the Analog Devices papers,118529.msg892562.html#msg892562?

They describe evaluation of ADCs.

At least read this one

If you want to get serious about understanding issues with ADCs, Walt Kester has many good articles.

Here is a link to his Handbook on Data Conversion  It covers both ADC and DAC issues.
943  Using Arduino / Storage / Re: Unformatted write to SD on: August 16, 2012, 10:52:03 am
Writing with no FS will not help.  Avoiding some block will not help. 

You can't solve the problem by speculation.  Remember, "One fact is worth more than a 1000 speculations".

The SD standard has lots of alignment requirements for how to format the SD so that performance will be optimal.  If you use the SD Association's formatter or my formatter, the SdFat SdFormatter.ino example, file structures will be properly aligned with erase groups.

You can't guess what the best policy will be.  I have spent days trying and every card is different and the behavior varies with card use.

Only two things seem matter and the big thing is the SD card controller.  If you have a good controller, you must use multi-block write and selecting write with pre-erase seems to help.

I added the ability to quickly create a large contiguous file to SdFat.  Doing raw writes to these files is just as good as having no FS and access is easier on other computers.  You have more flexibility than using something like dd.  Multiples regions on one SD become a pain with dd.

Unfortunately cards with really good performance are no longer being manufactured.  Cards that look the same have different controllers.

My best card was manufactured in 2007 and is a 2GB SanDisk Extreme III.  This is version 8.0 of this model card.

The standard SD card (cards with 2GB or less) are being phased out. 

Some SDHC cards perform fairly well with Arduino but again cards of the same model vary depending on the card version.

I have had good luck with some 4GB SanDisk Extreme cards.

Here are two examples, a 2GB card and a 4GB card.  Notice that block groups on the 2GB card are much smaller than the 4GB card.  32 blocks vs 128 blocks.  Also alignment of the FS partition on the 4GB card has a big unused space before the partition.  The 2GB card has a smaller space before the FAT partition.

2GB Extreme III card:
Manufacturer ID: 0X3
Product: SD02G
Version: 8.0
Serial number: 395023392
Manufacturing date: 11/2007

cardSize: 3970048 (512 byte blocks)
flashEraseSize: 32 blocks
eraseSingleBlock: true

SD Partition Table
4GB Extreme HD Video card:
Manufacturer ID: 0X3
Product: SD04G
Version: 8.0
Serial number: 3027274498
Manufacturing date: 4/2011

cardSize: 7744512 (512 byte blocks)
flashEraseSize: 128 blocks
eraseSingleBlock: true

SD Partition Table
944  Using Arduino / Storage / Re: Unformatted write to SD on: August 15, 2012, 05:23:41 pm
It's from the spec and experiment.  I was shocked years ago when I wrote my first SD library.

Actually if the maximum write latency is only 100 ms you are lucky.  Here is the ugly spec:
Quote Write
For a Standard Capacity SD Memory Card, the times after which a timeout condition for write operations occurs are (card independent) either 100 times longer than the typical program times for these operations given below or 250 ms (the lower of the two). The R2W_FACTOR field in the CSD is used to calculate the typical block program time obtained by multiplying the read access time by this factor. It applies to all write commands (e.g. SET(CLR)_WRITE_PROTECT, PROGRAM_CSD and the block write commands).

High Capacity SD Memory Card and Extended Capacity SD Memory Card indicate R2W_FACTOR as a fixed value.  In case of High Capacity SD Memory Card, maximum length of busy is defined as 250ms for all write operations.

So even if the card has been erased, the spec allowed a card to have long busy periods occasionally while programming flash.  

I have very little information what the cards controller is doing during this time.  Most of the details about the card controller are a manufacture's trade secret.  I suspect it is due to a wear-leveling operation.  the card won't program the block even though it has been erased.  wear-leveling happens for very large areas like 128 KB.  This requires a huge copy.  In streaming mode you tell the card what blocks will be written so the controller can plan ahead.

I have experimented with about 40 cards and find that many standard (2GB or less) cards maintain low write latency if you use multi-block streaming mode and space writes at even intervals.  

Many better SanDisk cards perform well in this mode.  Here is a benchmark at 500 blocks per second for a 2GB Extreme III card:
Start raw write of 5120000 bytes at
256000 bytes per second
Please wait 20 seconds
Elapsed time: 20000 millis
Max write time: 828 micros
Overruns: 0
The max time for a 512 byte write was 828 usec.

Block mode write and SPI are available for compatibility with the spec but may have very poor performance.

SD card are designed to be cheap and assume that devices like video cameras have a large amount of buffering so occasional long write latency is OK.  Class 10 card performance is based on the average rate for writing many MB of data.

The new SDXC cards can be busy for up to 500 ms.
945  Using Arduino / Storage / Re: writing time on SD on: August 15, 2012, 03:03:59 pm

No A RAM disk is not what I want. 

I want the ADCs (multiple ADCs) integrated with the FIFO and controlled by a CPLD or FPGA. 

I want to be able to program the sample rate so samples are taken at precise intervals and clocked into the FIFO without intervention by the MCU.  This would allow very low sample interval jitter.

The MCU reads from the FIFO, maybe does some processing, and writes the data to storage.

Analog Devices makes a board with this functionality for high speed ADC testing and evaluation but it is very high cost.  The Analog board can do 133 million samples per second.
Pages: 1 ... 61 62 [63] 64 65 ... 109