Show Posts
Pages: 1 ... 63 64 [65] 66 67 ... 111
961  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 03:14:57 pm
Wear leveling algorithms are not always a totally black box.  Look at this ]].

Tell me how it helps if you can't access the use counts and mappings.

Since every manufacturer has different internal structures and algorithms it is even harder.
962  Using Arduino / Storage / Re: Unformatted write to SD on: August 17, 2012, 01:50:38 pm
It seems that the potential 100ms busy time is a bit of a killer for what we are both trying to do.
No, I am perfectly happy with the result I posted above.  Here it is again:
Start raw write of 5120000 bytes at
256000 bytes per second
Please wait 20 seconds
Elapsed time: 20000 millis
Max write time: 828 micros
Overruns: 0
This means I can write at up to 256 KB/sec and the time to write a block is no greater than 828 usec.  There is no busy delay so about 42% of the CPU time is required.  This program simulates a data logger by writing a block every 2,000 usec in the multi-block mode I described before.

It is very difficult to write 512 bytes from RAM to the SPI bus much faster.

For typical logging applications, most of the CPU time is used acquiring data.  Writing at 100 KB/sec requires less than 20% of the CPU so 80% is available to acquire data.

The above result is for raw writes to a large contiguous file.  The fact the SD is formatted with a file system has no effect.

If you use the same SD for logging with single block writes to a file you get this result for 100 byte writes:
Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 199.21 KB/sec
Maximum latency: 86384 usec, Minimum Latency: 84 usec, Avg Latency: 496 usec
There was at least on busy delay of 86.4 ms with this SD.  The minimum latency, 84 usec, occurs when the write is just a copy to the SdFat block buffer and no write to the SD occurs.

This is one of the best SD cards around for Arduino use and it has almost a 100 ms delay.

The rate was under 200 KB/sec and required 100% CPU.

So what do you expect to achieve? 

Why do you want to use single block writes.  The above test proves that streaming multi-block writes work with good cards.
963  Using Arduino / Storage / Re: Logging 100 ksps with the Arduino internal ADC on: August 16, 2012, 11:04:53 am
The papers are included in  There are two tests of the AVR ADC.

Have you read the Analog Devices papers,118529.msg892562.html#msg892562?

They describe evaluation of ADCs.

At least read this one

If you want to get serious about understanding issues with ADCs, Walt Kester has many good articles.

Here is a link to his Handbook on Data Conversion  It covers both ADC and DAC issues.
964  Using Arduino / Storage / Re: Unformatted write to SD on: August 16, 2012, 10:52:03 am
Writing with no FS will not help.  Avoiding some block will not help. 

You can't solve the problem by speculation.  Remember, "One fact is worth more than a 1000 speculations".

The SD standard has lots of alignment requirements for how to format the SD so that performance will be optimal.  If you use the SD Association's formatter or my formatter, the SdFat SdFormatter.ino example, file structures will be properly aligned with erase groups.

You can't guess what the best policy will be.  I have spent days trying and every card is different and the behavior varies with card use.

Only two things seem matter and the big thing is the SD card controller.  If you have a good controller, you must use multi-block write and selecting write with pre-erase seems to help.

I added the ability to quickly create a large contiguous file to SdFat.  Doing raw writes to these files is just as good as having no FS and access is easier on other computers.  You have more flexibility than using something like dd.  Multiples regions on one SD become a pain with dd.

Unfortunately cards with really good performance are no longer being manufactured.  Cards that look the same have different controllers.

My best card was manufactured in 2007 and is a 2GB SanDisk Extreme III.  This is version 8.0 of this model card.

The standard SD card (cards with 2GB or less) are being phased out. 

Some SDHC cards perform fairly well with Arduino but again cards of the same model vary depending on the card version.

I have had good luck with some 4GB SanDisk Extreme cards.

Here are two examples, a 2GB card and a 4GB card.  Notice that block groups on the 2GB card are much smaller than the 4GB card.  32 blocks vs 128 blocks.  Also alignment of the FS partition on the 4GB card has a big unused space before the partition.  The 2GB card has a smaller space before the FAT partition.

2GB Extreme III card:
Manufacturer ID: 0X3
Product: SD02G
Version: 8.0
Serial number: 395023392
Manufacturing date: 11/2007

cardSize: 3970048 (512 byte blocks)
flashEraseSize: 32 blocks
eraseSingleBlock: true

SD Partition Table
4GB Extreme HD Video card:
Manufacturer ID: 0X3
Product: SD04G
Version: 8.0
Serial number: 3027274498
Manufacturing date: 4/2011

cardSize: 7744512 (512 byte blocks)
flashEraseSize: 128 blocks
eraseSingleBlock: true

SD Partition Table
965  Using Arduino / Storage / Re: Unformatted write to SD on: August 15, 2012, 05:23:41 pm
It's from the spec and experiment.  I was shocked years ago when I wrote my first SD library.

Actually if the maximum write latency is only 100 ms you are lucky.  Here is the ugly spec:
Quote Write
For a Standard Capacity SD Memory Card, the times after which a timeout condition for write operations occurs are (card independent) either 100 times longer than the typical program times for these operations given below or 250 ms (the lower of the two). The R2W_FACTOR field in the CSD is used to calculate the typical block program time obtained by multiplying the read access time by this factor. It applies to all write commands (e.g. SET(CLR)_WRITE_PROTECT, PROGRAM_CSD and the block write commands).

High Capacity SD Memory Card and Extended Capacity SD Memory Card indicate R2W_FACTOR as a fixed value.  In case of High Capacity SD Memory Card, maximum length of busy is defined as 250ms for all write operations.

So even if the card has been erased, the spec allowed a card to have long busy periods occasionally while programming flash.  

I have very little information what the cards controller is doing during this time.  Most of the details about the card controller are a manufacture's trade secret.  I suspect it is due to a wear-leveling operation.  the card won't program the block even though it has been erased.  wear-leveling happens for very large areas like 128 KB.  This requires a huge copy.  In streaming mode you tell the card what blocks will be written so the controller can plan ahead.

I have experimented with about 40 cards and find that many standard (2GB or less) cards maintain low write latency if you use multi-block streaming mode and space writes at even intervals.  

Many better SanDisk cards perform well in this mode.  Here is a benchmark at 500 blocks per second for a 2GB Extreme III card:
Start raw write of 5120000 bytes at
256000 bytes per second
Please wait 20 seconds
Elapsed time: 20000 millis
Max write time: 828 micros
Overruns: 0
The max time for a 512 byte write was 828 usec.

Block mode write and SPI are available for compatibility with the spec but may have very poor performance.

SD card are designed to be cheap and assume that devices like video cameras have a large amount of buffering so occasional long write latency is OK.  Class 10 card performance is based on the average rate for writing many MB of data.

The new SDXC cards can be busy for up to 500 ms.
966  Using Arduino / Storage / Re: writing time on SD on: August 15, 2012, 03:03:59 pm

No A RAM disk is not what I want. 

I want the ADCs (multiple ADCs) integrated with the FIFO and controlled by a CPLD or FPGA. 

I want to be able to program the sample rate so samples are taken at precise intervals and clocked into the FIFO without intervention by the MCU.  This would allow very low sample interval jitter.

The MCU reads from the FIFO, maybe does some processing, and writes the data to storage.

Analog Devices makes a board with this functionality for high speed ADC testing and evaluation but it is very high cost.  The Analog board can do 133 million samples per second.
967  Using Arduino / Storage / Re: writing time on SD on: August 15, 2012, 10:08:22 am

The fast external RAM is an old idea and won't help much for fast data acquisition.

Here is a more interesting use for a large RAM.  Use the RAM as a FIFO for a really fast ADC.  You could clock data into the RAM from the ADC at 15 MB/sec or faster and read it out over SPI at Arduino speeds.  Even 4 MB at several million samples per second would be interesting.

Interfacing several ADC to the RAM FIFO would be even more interesting.  Then you would have parallel channels.

Adding a fast external ADC to an Arduino doesn't work very well since it's hard to avoid excessive jitter in the sample interval.

Some internal clocked ADC have been used with an Arduino and triggered by a timer compare signal on a pin but these are not really fast.

In many application, jitter is as important as ADC accuracy.  If you don't know when a measurement was taken you haven't captured the signal wave form.

Edit: Reading the FIFO in parallel with about 11 pins would be fine.
968  Using Arduino / Storage / Re: Unformatted write to SD on: August 14, 2012, 04:30:13 pm
Writing to the SD is not as much the problem as acquiring the data.  

You must use multi-block streaming write commands.  I have posted many sketches and there is a RawWrite example in my latest version of SdFat.

Data is worthless for most measurements unless it is taken at precise time intervals.

The quality of a logger is often determined by logging a pure sine wave and then doing a FFT.  You won't get good results if there is time jitter or too much noise.

Effective number of bits (ENOB) is derived from an FFT analysis of the ADC output when the ADC is stimulated with a full-scale sine-wave input signal. The root-sum-of-squares (RSS) value of all noise and distortion terms is computed, and the ratio of the signal to the noise-and-distortion is defined as SINAD, or S/(N+D).

I suggest more thought on acquisition.
969  Using Arduino / Storage / Re: Logging 100 ksps with the Arduino internal ADC on: August 14, 2012, 04:20:15 pm
ENOB (not ENOD) has a very precise meaning. 

I included two papers with the fast logging sketch that describe how it is determined for the AVR ADC.
970  Using Arduino / Storage / Re: writing time on SD on: August 14, 2012, 04:05:21 pm
32 KB and even FRAM/MRAM is really limiting for many fast logging apps. 

I log 16 bit data at 120 KB/sec so it eats storage.
971  Using Arduino / Storage / Re: Unformatted write to SD on: August 14, 2012, 03:51:11 pm
You must write an entire block at a time.  You must send all 512 bytes of data and two dummy or CRC bytes in one SPI transfer.

You can't implement a fast logger this way since the SD may present a busy delay of up to 100 ms when you send a single block write command.

Only streaming block write commands are fast.  See  It can do 100,000 samples per second.

You can't log faster with an external ADC since you can't used the SPI bus with the ADC and SD.  The SD requires the block write to be done as a single SPI transfer.

I have a very fast Software SPI routine but it only runs at 2MHz so about 40,000 samples per second is the limit for the sketch I posted at the above site for fast logging with an external ADC.
972  Using Arduino / Storage / Re: writing time on SD on: August 14, 2012, 03:31:05 pm
Much of what ardnut says doesn't help.  The item that is correct is SD controllers do load leveling so you won't damage a block by writing it repeatedly.

It is not using the SD as a block device that produces low latency writes.  Single block writes are very unpredictable in terms of latency.

Erasing all blocks doesn't help much in single block write mode.  I erase all blocks in my SdFat formatting sketch but single block mode still has long latencies on a freshly formatted SD.

To write with low latency, you must use multi-block streaming commands.  There is a streaming command with pre-erase.

I just posted a sketch that logs data from an analog pin at up to 100,000 samples per second without losing data.  This sketch uses streaming writes to a preallocated contiguous file. 

This logger has very low time jitter in the interval between samples, a few ns at most.  It uses a timer compare event to trigger the ADC which is much better than using a timer interrupt.

The sketch is in
973  Using Arduino / Storage / Re: Logging 100 ksps with the Arduino internal ADC on: August 12, 2012, 02:00:22 pm
Yes you can buy faster hardware.  Compared to the area of science I came from your new chip is molasses.

If it isn't a flash ADC in the GHz range it's slow.  Big experiments also handle incredible amounts of data.

The Atlas experiment at CERN discovered the Higgs Boson.  Atlas digitizes 40,000,000 events per second and each event has 1.6 MB of raw data.  That's 64,000,000,000,000 bytes per second. 

This data is processed by fast hardware in real time and most is discarded.  Atlas only records 320 MB/sec, one event in 200,000.

The first level processing has 72 racks like this:

I record 100 KB/sec in my sketch, a factor of 3,200 less.

All of this little use to the average hobbyist.  I find few takers for my fast AVR logging programs.

The typical email I get asking about fast logging is for something like an accelerometer with three channels at 1000 samples per second.  Most hobbyists don't realize that digitizing all channels of an event at the same time and at precise time intervals is key to good data.

Here many of the STM32 chips have something to offer.  Some of the STM32 chips have three ADCs that can be trigger at the same time.

The single ADC in AVR chips make it impossible to digitize multiple channels properly.  You can use several self-clocking external ADCs to do much better.

I like playing with the AVR because it is a challenge to do good measurements.  I did a lot of work on this sketch to reduce jitter and signal to noise.

I start the ADC on a timer compare event, not an interrupt.  I make sure the timer triggers at a multiple of the ADC clock period.  This reduces time jitter.  I use DIDR to disconnect the pin's digital input.  An analog input floats all over the place, and causing the digital input to constantly toggle high and low. This creates excessive noise near the ADC.
974  Using Arduino / Storage / Re: Can't load entire contents of SD card on: August 12, 2012, 10:09:10 am
It may seem silly but you still need the break for all the other cases.

Making readToken() elegant could be an interesting exercise in programming.

975  Development / Other Hardware Development / Re: New Olimex STM32-E407 on: August 11, 2012, 01:20:54 pm
I now see the Olimex STM32-E407 on ebay.  

I decided to buy an Olimex STM32-H407 instead since it costs less but has what I need.

I got one on ebay with 5-7 business day shipping to the U.S. for £27.95 GBP or $44.99 with PayPal's 2.5% extra exchange rate.

Here is the difference between boards:
The major difference between STM32-H407 and STM32-E407 is that the latter has built-in Ethernet (physical level transceiver Micrel - Micrel datasheet). STM32-E407 also features an extra USB-OTG and a number of SMD jumpers on the bottom which help the user to control the multiplexing on some pins easier. STM32-E407 has 2 x USB-OTG both with a miniUSB interface. STM32- H407 has 1 x USB-OTG and 1 x USB-HOST with the On-The-Go interfaced by miniUSB and the HOST by USB type A connector.

The STM32-E407 is £10.00 GBP more.
Pages: 1 ... 63 64 [65] 66 67 ... 111