Show Posts
Pages: 1 ... 61 62 [63] 64 65 ... 111
931  Using Arduino / Storage / Re: Unformatted write to SD on: August 30, 2012, 09:40:04 am
Thanks, I meant to check if the compile order was correct for 16-bit access but never got back to it.  
932  Using Arduino / Storage / Re: High Performance Industrial SD Cards on: August 28, 2012, 06:24:37 pm
Here is a low cost high performance card.

An ATP 1GB Secure Digital SLC Industrial Grade card for $5.75 when I bought it.

http://www.newegg.com/Product/Product.aspx?Item=9SIA12K0CT6829

Here are the SdFat bench results for 100 and 512 byte records.
Quote
Buffer size 100 bytes
Write 258.95 KB/sec
Maximum latency: 69400 usec, Minimum Latency: 84 usec, Avg Latency: 380 usec
Read 329.38 KB/sec
Maximum latency: 2240 usec, Minimum Latency: 80 usec, Avg Latency: 298 usec


Buffer size 512 bytes
Write 392.32 KB/sec
Maximum latency: 62500 usec, Minimum Latency: 1184 usec, Avg Latency: 1339 usec
Read 469.41 KB/sec
Maximum latency: 2164 usec, Minimum Latency: 1072 usec, Avg Latency: 1085 usec
933  Using Arduino / Storage / Re: SDFat - pulling out the card on: August 28, 2012, 06:04:14 pm
Sorry you are so upset.  I thought it was strange that you assumed an Ethernet shield was used.
Quote
Guess you have the ethernet shield if you have the SD reader.

At least a thousand people have contacted me about data logging and only a small percentage use the Ethernet shield for data logging.

Sorry, I can't refer you to a peer-reviewed paper.
Quote
Wondering what your research method was: group size, demographic, distribution, duration, correlation ...
You may want to direct me to the peer-reviewed paper you have written on the subject.

Shields like this one from Adafruit are very popular http://www.adafruit.com/products/243.

Many people use the SparkFun shield https://www.sparkfun.com/products/9802.

People like these shields for data logging since they have a prototyping area.

Networking is a good solution and a number of people have used Ethernet to upload data from an SD.

The WiFi Shield can be used http://arduino.cc/en/Main/ArduinoWiFiShield.

Another Idea people are using is an Eye Fi card http://www.ladyada.net/make/IoTcamera/.

934  Using Arduino / Storage / Re: SDFat - pulling out the card on: August 28, 2012, 08:58:10 am
Most people who log data to an SD do not use the Ethernet Shield.  There are now dozens of SD shields/modules.

A network is great if it is suitable.
935  Using Arduino / Storage / Re: Runtime determination of EEPROM size on: August 27, 2012, 04:38:53 pm
The symbol E2END is the last EEPROM address.

For a 328 Arduino this sketch
Code:
void setup() {
  Serial.begin(9600);
  Serial.println(E2END);
}
void loop() {}

Prints this

Quote
1023

See this http://www.nongnu.org/avr-libc/user-manual/group__avr__io.html
936  Using Arduino / Storage / Re: micro SD "HC class 4" card on: August 27, 2012, 08:03:07 am
Here are the results for a Class 4 Kingston 4GB HC card with the Ethernet shield on an UNO.

http://www.amazon.com/Kingston-microSDHC-Memory-SDC4-4GBET/dp/B00200K1TI

For SdInfo:
Quote
init time: 990 ms

Card type: SDHC

Manufacturer ID: 0X41
OEM ID: 42
Product: SD4GB
Version: 2.0
Serial number: 2711490139
Manufacturing date: 3/2009

cardSize: 7745536 (512 byte blocks)
flashEraseSize: 128 blocks
eraseSingleBlock: true

SD Partition Table
part,boot,type,start,length
1,0X0,0XB,8192,7737344
2,0X0,0X0,0,0
3,0X0,0X0,0,0
4,0X0,0X0,0,0

Volume is FAT32
blocksPerCluster: 64
clusterCount: 120768
freeClusters: 120539
fatStartBlock: 14496
fatCount: 2
blocksPerFat: 944
rootDirStart: 2
dataStartBlock: 16384

Here is the SdFat bench result (Note write is very slow):
Quote
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 65.34 KB/sec
Maximum latency: 234516 usec, Minimum Latency: 84 usec, Avg Latency: 1521 usec

Starting read test.  Please wait up to a minute
Read 278.91 KB/sec
Maximum latency: 3588 usec, Minimum Latency: 80 usec, Avg Latency: 353 usec

Here are the bench results for a Class 10 Amazon 4GB card http://www.amazon.com/AmazonBasics-microSDHC-Class-Memory-Adapter/dp/B0058GH3II
Quote
Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 169.04 KB/sec
Maximum latency: 189888 usec, Minimum Latency: 84 usec, Avg Latency: 586 usec

Starting read test.  Please wait up to a minute
Read 288.37 KB/sec
Maximum latency: 3136 usec, Minimum Latency: 80 usec, Avg Latency: 341 usec
937  Using Arduino / Storage / Re: High Performance Industrial SD Cards on: August 26, 2012, 02:53:18 pm
The next generation of consumer SD cards will be even slower on Arduino.

Here is the result of running bench on a 64GB SDXC card.

Quote
Write 116.69 KB/sec
Maximum latency: 192708 usec, Minimum Latency: 84 usec, Avg Latency: 851 usec
Read 296.70 KB/sec
Maximum latency: 2980 usec, Minimum Latency: 80 usec, Avg Latency: 331 usec

The reason is that new cards have very dense TLC (Triple Level Cell) NAND flash.  Three bits are stored in the gate of the one transistor cell as eight levels of charge, each level differing by less than 100 electrons.

These chips are designed for very large contiguous writes.  The Arduino does not have sufficient buffering to achieve this.

The page size of these chips is much larger than a 512 byte block so in only part of a page is used when a single 512 byte block is written. The rest of the page is "dead" meaning unusable until it is erased.

Eventually the SD controller collect these blocks and rewrites them to a new page.  This is a very high overhead process.

In addition TLC flash is very susceptible to wear so the Arduino's inefficient use of the device causes lots of data movement.  The Erase Block Size for these flash chips can be 256KB. Moving this much data take a long time.

Even unchanged data gets moved with the wear-leveling algorithms.
Quote
Blocks that contain static data with erase counts that begin to lag behind other blocks will be included in the wear-leveling block pool, with the static data being moved to blocks with higher erase counts.


The 64GB SDXC card I tested can have an occasional write latency of almost 200 ms.

If you want to learn more about the internals of SD cards and modern NAND flash here are some links.

http://www.micron.com/~/media/Documents/Products/Technical%20Note/NAND%20Flash/151tn2942_nand_wear_leveling.pdf

http://www.es.ele.tue.nl/~kgoossens/2011-codes+isss-special-session/kuo.pdf

http://www.eetindia.co.in/STATIC/PDF/200809/EEIOL_2008SEP22_STOR_AN_01.pdf
938  Using Arduino / Storage / Re: sd with serial communication, how do? on: August 26, 2012, 02:21:31 pm
There is a serious bug in the dynamic memory free() function that can cause String to crash.

http://arduino.cc/forum/index.php/topic,115552.0.html

Fixes are presented in the above topic.

The problem may also cause SD.h to crash if many files are opened an closed.
939  Using Arduino / Storage / High Performance Industrial SD Cards on: August 25, 2012, 02:49:25 pm
Many new consumer SD cards have poor file write performance with Arduino SD libraries.

I bought and tested two industrial grade SD cards and compared them with two consumer SD cards.  Industrial SD cards cost a great deal more than consumer cards.

The industrial cards are:

Swissbit 1GB card http://www.digikey.com/product-detail/en/SFSD1024L1BN2TO-I-ME-151-STD/1052-1033-ND/2620367.

Wintec 1GB card http://www.digikey.com/product-detail/en/W7SD001G1XA-H60PD-002.02/385-1079-ND/3152960

The consumer cards are:

4GB Amazon class 10 card http://www.amazon.com/AmazonBasics-SDHC-Class-Secure-Digital/dp/B004Q3R9AQ/ref=sr_1_3?ie=UTF8&qid=1345922948&sr=8-3&keywords=4gb+class+10+sdhc

2GB SanDisk class 2 card http://www.amazon.com/SanDisk-Class-Flash-Memory-SDSDB-2048-A11/dp/B0009RGLSE/ref=sr_1_8?s=electronics&rps=1&ie=UTF8&qid=1345923062&sr=1-8&keywords=2gb+sandisk+class+2+sd

I ran the SdFat bench example with two record sizes.  First I used the default 100 byte records.  Here is a summary of write results.
Quote
1GB Wintec: 278.40 KB/sec
1GB Swissbit: Write 265.10 KB/sec
2GB SanDisk class 2: 127.41 KB/sec
4GB Amazon class 10: 165.69 KB/sec

Next I ran the bench example with 512 byte records.  In the case of 512 byte records, data is written directly to the SD without being copied to SdFat's internal buffer.   Here is a summary of write results.
Quote
1GB Wintec: 407.57 KB/sec
1GB Swissbit: 384.92 KB/sec
2GB SanDisk class 2: 152.31 KB/sec
4GB Amazon class 10: 200.62 KB/sec

In this case the industrial cards are about twice as fast as the consumer cards.  There is not as much gain for read.

Here are the detailed results.
Quote
100 byte records

1GB Wintec card
Write 278.40 KB/sec
Maximum latency: 84916 usec, Minimum Latency: 84 usec, Avg Latency: 354 usec
Read 338.89 KB/sec
Maximum latency: 2164 usec, Minimum Latency: 80 usec, Avg Latency: 289 usec

1GB Swissbit card
Write 265.10 KB/sec
Maximum latency: 72608 usec, Minimum Latency: 84 usec, Avg Latency: 372 usec
Read 338.07 KB/sec
Maximum latency: 2172 usec, Minimum Latency: 80 usec, Avg Latency: 290 usec

2GB Sandisk class 2 card
Write 127.41 KB/sec
Maximum latency: 61372 usec, Minimum Latency: 84 usec, Avg Latency: 778 usec
Read 276.33 KB/sec
Maximum latency: 3180 usec, Minimum Latency: 80 usec, Avg Latency: 356 usec

4GB Amazon class 4 card
Write 165.69 KB/sec
Maximum latency: 191320 usec, Minimum Latency: 84 usec, Avg Latency: 598 usec
Read 290.17 KB/sec
Maximum latency: 3108 usec, Minimum Latency: 80 usec, Avg Latency: 339 usec

512 byte records

1GB Wintec card
Write 407.57 KB/sec
Maximum latency: 85756 usec, Minimum Latency: 1152 usec, Avg Latency: 1250 usec
Read 488.39 KB/sec
Maximum latency: 2088 usec, Minimum Latency: 1032 usec, Avg Latency: 1043 usec

1GB Swissbit card
Write 384.92 KB/sec
Maximum latency: 67700 usec, Minimum Latency: 1160 usec, Avg Latency: 1324 usec
Read 486.68 KB/sec
Maximum latency: 2096 usec, Minimum Latency: 1036 usec, Avg Latency: 1046 usec

2GB SanDisk class 2 card
Write 152.31 KB/sec
Maximum latency: 59452 usec, Minimum Latency: 2132 usec, Avg Latency: 3355 usec
Read 366.17 KB/sec
Maximum latency: 2700 usec, Minimum Latency: 1288 usec, Avg Latency: 1393 usec

4GB Amazon class 10 card
Write 200.62 KB/sec
Maximum latency: 191596 usec, Minimum Latency: 2132 usec, Avg Latency: 2545 usec
Read 391.00 KB/sec
Maximum latency: 3020 usec, Minimum Latency: 1164 usec, Avg Latency: 1304 usec

940  Using Arduino / Sensors / Re: Static Accuracy Tests of the Arduino Internal ADC. on: August 25, 2012, 11:24:49 am
Both WaveRP and AnalogIsrLogger have configurable sample rates.  You need to look at the programs and documentation for more details.
941  Using Arduino / Sensors / Re: Static Accuracy Tests of the Arduino Internal ADC. on: August 25, 2012, 06:48:53 am
There is documentation in the zip files.

I originally designed SdFat for recording audio and fast data logging.  I added features similar to the POSIX real-time extensions for files.

Large contiguous files can be created quickly.  There is a method to do raw writes to the SD using an efficient multi-block mode.  When a file is closed, the unused space can be truncated.

This allows very low latency writes to high quality SD cards.  The best cards are standard cards (2 GB or less) such as the SanDisk Extreme III.  Manufacturers have stopped producing many of these consumer grade cards so I am now experimenting with industrial grade cards like those made by Swissbit.

The write time for a 512 byte block is about 820 usec with good cards.

I have had some luck with SDHC cards like the 4GB and 8 GB SanDisk Extreme cards but manufacturers assume devices that use these cards have lots of RAM for buffering and can tolerate an occasional long write latency.

Download the 100,000 sample per second logger for an example that uses of these features.
942  Using Arduino / Sensors / Re: Static Accuracy Tests of the Arduino Internal ADC. on: August 24, 2012, 08:07:17 pm
It is possible to record 8-bit 44 ksps audio at PS16. Voice is not bad, like a cheap recorder.  I wrote a library to do it, WaveRP, http://code.google.com/p/waverp/downloads/list.

I also did a 8-bit 100,000 sample per second logger, AnalogIsrLogger20120810.zip http://code.google.com/p/beta-lib/downloads/list.  The Arduino gets an ENOB of about 7.5 at this speed.  You need a low impedance source.  I did a lot of work to reduce time jitter.

MarkT,

These tests are not appropriate for dynamic ADC performance. 

Here is a good article http://www.openmusiclabs.com/learning/digital/atmega-adc/.

You just can't use the AVR Arduino at high sample rates with high impedance sources.  If the ADC clock is 1MHz, the sample and hold time is on the order of a microsecond.

I like to have 10 time constant, RC, periods to charge the sample and hold cap.  10 K Ohms is just too large.

Calibration would change with temperature since gain and offset change.

Integral Nonlinearity is usually less dependent on temperature.

Atmel doesn't publish much about the properties of the AVR ADC so you can't be sure without tests.  Doing tests of ADC properties as a function of temperature for the AVR ADC would be a pain.

The Arduino ADC is not bad for most hobby use with low impedance sources.
943  Using Arduino / Sensors / Static Accuracy Tests of the Arduino Internal ADC. on: August 24, 2012, 01:57:39 pm
Several people have asked about the DC accuracy of the Arduino ADC when used in my data logging applications at slow sample rates.

Here are my results of some "hobby level" measurements of the Arduino ADC.

One question is how important is the ADC clock rate.  I did measurements for an ADC clock rate of 125 kHz to 2MHz.

Another question is how much does Noise Reduction Mode help.  I did a series of measurements using this mode.

Noise Reduction Mode only reduced the mean absolute error slightly.

I do calibration to remove Offset Error and Gain Error.  Calibration is very important for good accuracy.

These tests depend on the Arduino voltage regulator providing a stable voltage during the tests.  The Arduino ADC reference voltage is Vcc for these tests.  This may not be realistic for practical applications

Integral Non-linearity (INL) is the main remaining source of error.

Here are my results for static (DC) tests of the internal ADC for three UNOs.

The Arduinos are powered by a high quality nine volt power supply.

These tests measure a DC level so do not include problems due to time jitter, S/H time, and other dynamic errors.
There are several studies of the dynamic behavior of the Arduino ADC that determine ENOB (Effective Number Of Bits).

I used a shield with a 12-bit MCP4921 DAC to generate voltage levels. This DAC has an output buffer so it provides a very low impedance source.

I measured the voltage of the DAC with a calibrated 18-bit MCP3422 ADC on the shield.

I used DAC levels from 20 to 4075 to avoid zero offset errors at low voltages and DAC buffer problems at high voltages.

Each series of measurements has 4056 data points.

This is a voltage range of about 0.023 to 4.972 volts.

I calibrated the Arduino ADC for each series of measurements with a linear fit of the form.

v = a + b*adcValue

Errors are the difference between the value measured with the 18-bit ADC and the calibrated value measured with the Arduino ADC.

I also show the results for no calibration, the NoCal column, using the datasheet formula.

Vin = Vref*adcValue/1024


The columns in the tables are:

Ideal - results for a perfect 10-bit ADC for comparison.

NoCal - datasheet formula (5/1024)*adcValue with Noise Reduction Mode.

NR128 - Noise Reduction mode with Prescaler of 128 (ADC clock of 125 kHz).

PS64 - analogRead with Prescaler of 128 (ADC clock of 125 kHz).

PS64 - analogRead with Prescaler of 64 (ADC clock of 250 kHz).

PS32 - analogRead with Prescaler of 32 (ADC clock of 500 kHz).

PS16 - analogRead with Prescaler of 16 (ADC clock of 1 MHz).

PS8 - analogRead with Prescaler of 8 (ADC clock of 2 MHz).


The rows in the tables tables are.

Min - minimum error in millivolts

Max - maximum error in millivolts

MAE - mean absolute error in millivolts


Results for three UNO Arduinos (sorry about the ragged tables, I haven't mastered table insertion)

    First Arduino - Error Millivolts
      
       Ideal   NoCal  NR128  PS128   PS64   PS32   PS16    PS8
Min   -2.44  -2.43    -3.72   -4.01  -3.88  -4.53  -6.57  -27.18
Max   2.44  11.69     3.74    4.24    4.15   5.17   8.69   23.21
MAE   1.22   5.02     1.33    1.38    1.37   1.44   1.96     4.11

     Second Arduino - Error Millivolts

       Ideal   NoCal  NR128  PS128   PS64   PS32   PS16    PS8
Min   -2.44  -9.24    -4.87   -4.86  -5.05  -5.34  -6.52  -24.04
Max   2.44  11.62     3.95    4.64    4.69   5.71    8.41   21.29
MAE   1.22   5.33      1.41    1.43   1.44   1.53     2.02    4.05

     Third Arduino - Error Millivolts

       Ideal   NoCal  NR128  PS128   PS64   PS32   PS16    PS8
Min  -2.44   -7.88    -4.12   -4.40   -4.32  -4.41  -6.97  -26.93
Max  2.44   12.53     3.80    4.04     4.18   5.27   8.84   24.59
MAE  1.22    4.85     1.29     1.33    1.34    1.42   1.91    4.10
944  Using Arduino / Storage / Re: 3.3v to 5v inverter by bc547 on: August 22, 2012, 12:49:43 pm
Resistors fail for some newer SD cards that have edge detectors. The SD specification requires clock rise time and fall time to be less than 10 ns.

Worse is the fact that the errors in write data won't be detected.  There is no CRC by default since Arduino has no hardware support.

The latest version of SdFat supports software CRC but this slows transfers.

945  Using Arduino / Storage / Re: Unformatted write to SD on: August 21, 2012, 10:06:39 am
Quote
But only if you are sampling within 3 LSB of the extremes of the dynamic range!
I don't think you meant that.  Here is the definition of dynamic range for an ADC:
Quote
Dynamic Range
Typically expressed in dB, dynamic range is defined as the range between the noise floor of a device and its specified maximum output level. An ADC's dynamic range is the range of signal amplitudes which the ADC can resolve; an ADC with a dynamic range of 60dB can resolve signal amplitudes from x to 1000x. Dynamic range is important in communication applications, where signal strengths vary dramatically. If the signal is too large, it over-ranges the ADC input. If the signal is too small, it gets lost in the converter's quantization noise.
I think you meant full scale of the ADC.  Here is the definition of ENOB:
Quote
Effective Number Of Bits (ENOB)
ENOB specifies the dynamic performance of an ADC at a specific input frequency and sampling rate. An ideal ADC's error consists only of quantization noise. As the input frequency increases, the overall noise (particularly in the distortion components) also increases, thereby reducing the ENOB and SINAD. (See 'Signal-to-Noise and Distortion Ratio (SINAD).') ENOB for a full-scale, sinusoidal input waveform is computed from:

ENOB = (SINAD -1.76)/6.02
Note full-scale.  If you use a smaller signal, you would get the wrong (smaller answer).

Want to bet on Higgs?   I may be wrong, not all properties have been verified.  On the other hand not all group information is public so you might want to think before you bet.  The correct decays have been seen but now the equivalent of oversampling is happening to be totally sure.  Hope you read the latest paper that has just been submitted to "Physics Letters B" when it's published. 

You need to lighten up a bit and so do I.  Here's how http://www.wikihow.com/Lighten-Up.  This is a summary:
Quote
1. Stop assuming you know everything. Nobody knows everything.

2. Stop exaggerating. Exaggerating about your abilities, qualifications, knowledge, hobbies etc. is soon tiresome.

3. Let go of things. It's OK to lose an argument; it's OK to make mistakes.

4. Laugh.

5. Delegate.

6. Stop being so rules focused.

You're welcome for the code and any information you can use.  I will try to lighten up also.
Pages: 1 ... 61 62 [63] 64 65 ... 111