Show Posts
Pages: 1 ... 54 55 [56] 57 58 ... 111
826  Using Arduino / Storage / Re: SD read -> I2C-DAC; latency and interrupt issues on: December 01, 2012, 09:42:12 am
There will always be reads that take over 1 ms so you need a solution for read that buffers ahead.

There are I2C libraries that don't use interrupts.  I wrote such a library that supports both hardware and software I2C but it is not well documented.  It is included in this file, AdafruitLogger20120609.zip  here http://code.google.com/p/beta-lib/downloads/list.  The library is I2cMaster.

You could look at my libraries for the Adafruit Wave Shield for examples that send data to an DAC at up to 44.1 ksps.

This is a player only library http://code.google.com/p/wavehc/downloads/list.  It uses a SPI DAC but could use a I2C DAC at slower speeds.

Here is a player/recorded library http://code.google.com/p/waverp/downloads/list.

These libraries solve the read ahead problem with a timer triggered thread.
827  Using Arduino / Storage / Re: Arduino Ethernet Shield microSD max size? on: November 30, 2012, 11:52:07 am
You can use any size SDHC card.  The max for SDHC is 32 GB.

The card class my not translate into performance.  Class 6 and 10 cards are optimized for large buffers and the high speed 4-bit SDIO bus.  These cards may not perform better than class 2 or 4 cards on the small memory Arduino with its slow SPI bus.   
828  Products / Arduino Due / Re: Happiness is SdFat with DMA SPI on: November 29, 2012, 02:46:06 pm
stimmer,

Thanks, I will probably put the Bus Matrix stuff in.  Maybe with conditionals so it's there if problems occur when I post this for test.

I was about to try the Bus Matrix next.

The SAM3X is new for me, I mostly work with STM32.

Configuring these things when your new feels like just try all possible combinations and something will work.  Trouble is the number of combination for n choose m is mighty large.  I tried both DMAC_CFG_FIFOCFG_ASAP_CFG and DMAC_GCFG_ARB_CFG_FIXED but not at the same time.
 
Edit: I have now run a lot of tests with 2 GB files using just the two DMAC changes.  I have probably have written and read 20 GB of data without an error.  It only takes a bit over eight minutes to read a 2 GB file.
829  Products / Arduino Due / Re: Happiness is SdFat with DMA SPI on: November 29, 2012, 12:28:00 pm
The return from spiRec are error bits, 0X2 means a timeout waiting for DMA to finish and 0X1 means SPI overrun was set.

I tried both changes with SdFat and it seems to work.  I will stress test with some long file reads.

Can't say I feel confident, wish I understood the Bus Matrix and DMAC controller better.

At least a test with 100 MB which is about 200,000 block reads ran O.K.
Quote
Type is FAT16
File size 100MB
Buffer size 81920 bytes
Starting write test.
Write 4019.24 KB/sec
Maximum latency: 83141 usec, Minimum Latency: 19453 usec, Avg Latency: 20376 usec

Starting read test.
Read 4387.86 KB/sec
Maximum latency: 18959 usec, Minimum Latency: 18598 usec, Avg Latency: 18667 usec
830  Using Arduino / Storage / Re: SD card latency help? on: November 29, 2012, 10:26:43 am
Chan's website is very good but most of the stuff about performance of SD cards is out of date. 

Wear leveling in modern TLC flash can dominate latency.  Modern cards move data that has not been changed for a long time to a new physical location.  Flash controllers have various amounts of RAM buffering and use different algorithms so generalizations like small cards are faster just isn't true.

Write latency also depends on the access pattern.  328 Arduinos have limited memory so there are not many caching options. 

Small cards are often formatted with small clusters which increases the overhead since the FAT must be accessed often.

For embedded applications cards with SLC flash tends to be best.  Modern consumer cards use MLC or TLC flash.  Industrial SD cards mostly use SLC flash.  I have had good luck with this cheap Industrial card http://www.newegg.com/Product/Product.aspx?Item=9SIA12K0CT6829.

No card has uniform low write latency.  All cards have occasional longer latencies.  You must design fast data loggers with this in mind to avoid data loss.

I am now working on improved performance for SdFat with Cortex chips and the Mega.  Here is a link to some results for the Due with DMA SPI http://arduino.cc/forum/index.php/topic,134512.0.html.
831  Products / Arduino Due / Re: Happiness is SdFat with DMA SPI on: November 29, 2012, 09:21:50 am
stimmer,

Edit: after posting this I changed RX FIFO handling to DMAC_CFG_FIFOCFG_ASAP_CFG and things seem to work better but still errors.

I attached a zip file with a test library and a sketch that demonstrates the problem (you must be logged in to see the attachment).

Put the DueSPI folder in libraries and run the due_spi sketch to see the problem.

I will be interested to see if changing Bus Matrix arbitration can make this work.


alvesjc,

You may be able to increase the speed of the Standard SD library.  I wrote the base code for the Standard SD library about three years ago so I have no interest in that old code.
832  Products / Arduino Due / Re: Happiness is SdFat with DMA SPI on: November 28, 2012, 03:25:57 pm
Looks like the problem with reads at 42 MHz only occurs if the data buffer for the read is in the top 32 KB of memory.

SRAM for the SAM3X is two banks, SRAM0 - 64 KB, and SRAM1 - 32 KB.  The stack is at the top of SRAM1 so when the interrupt for sysTick happens, registers are pushed on the stack in SRAM1.  This blocks access to SRAM1 by the DMA controller and I get an SPI overrun error.  If I disable interrupts during the 512 byte DMA transfer, no error occurs.  Too bad the SPI controller doesn't have a FIFO.  At 42 MHz, the SPI bus delivers a byte about every 200 ns.

It takes about 100 usec for a 512 byte transfer so it's too long to disable interrupts.

Most people will have their buffers in the first 64 KB of SRAM0 and access to to this bank works fine with 42 MHz SPI.  Also SPI at 28 MHz works with buffers in SRAM1.

DMA at 28 MHz is still fast:
Quote
Type is FAT16
File size 40MB
Buffer size 61440 bytes
Starting write test.  Please wait up to a minute
Write 2834.49 KB/sec
Maximum latency: 90293 usec, Minimum Latency: 20612 usec, Avg Latency: 21666 usec

Starting read test.  Please wait up to a minute
Read 3056.74 KB/sec
Maximum latency: 20401 usec, Minimum Latency: 19993 usec, Avg Latency: 20098 usec

The sending of 0XFF bytes to drive SPI clock have nothing to do with the problem.
833  Products / Arduino Due / Re: Happiness is SdFat with DMA SPI on: November 28, 2012, 07:47:28 am
pico,

You need DMA to get the speedup on Due.  There is no FIFO in the SPI controller.  The Teensy FIFO allows SPI to go at almost full speed, 24 MHz, without DMA.

For Due, I wrote non-DMA SPI optimized for SD cards and got about twice the standard SPI library speed.  The standard SPI library trades speed for features which is probably a good thing.

I will post this version of SdFat soon.  I need to do a lot more stress testing.

I have one puzzle, sometimes a DMA read hangs at 42 MHz.  To drive read, I use one DMA channel to send a stream of 0XFF bytes to the SPI controller.  Data is read from the SPI controller using a second DMA channel.  I sometimes get a hang when the 0XFF byte is in the same SRAM bank as the receive buffer.

I need to investigate options.  I am using byte transfers for SPI.  Other high speed interfaces like Ethernet and HSMCI can use 32-bit transfers. I am using a DMA channel for receive that has an eight deep FIFO.  I will try a channel with a 32 entry FIFO.
834  Using Arduino / Storage / Re: SD card latency help? on: November 27, 2012, 06:31:41 pm
Write latency is inherent to SD cards.  The SD spec allows up to 200 milliseconds latency for a write.  Occasional write latencies of 100 milliseconds are not uncommon.

You must design your logger to allow for latency.  I have posted several example fast loggers.

The cleanest way is to use a RTOS with two threads and read sensors in a high priority thread and buffer the data to be written to the SD in a lower priority thread.

I posted RTOS examples as ChibiOS20120529.zip and FreeRTOS20111031.zip here http://code.google.com/p/beta-lib/downloads/list.

Another way is to read the sensor in an ISR and buffer the data for loop.  AnalogIsrLogger20120810.zip posted in the above location is a very fast logger that reads the Arduino ADC in an ISR and write the the data in loop().

Several other demo sketches are in fastLoggerBeta20110802.zip.  This file is now a bit old.
835  Products / Arduino Due / Happiness is SdFat with DMA SPI on: November 27, 2012, 10:53:57 am
I decided to implement DMA SPI for SdFat on Due.  I couldn't be more pleased with the results!

Here are bench.ino results for very large, 32 KB, reads and writes with this class 4 Sandisk 8 GB card:

http://www.amazon.com/SanDisk-High-Capacity-Card-SDSDB-008G-B35/dp/B000VUVA62/

Quote
Type is FAT32

File size 40MB
Buffer size 32768 bytes
Starting write test.  Please wait up to a minute
Write 3835.82 KB/sec
Maximum latency: 45560 usec, Minimum Latency: 7815 usec, Avg Latency: 8538 usec

Starting read test.  Please wait up to a minute
Read 4324.64 KB/sec
Maximum latency: 12697 usec, Minimum Latency: 7495 usec, Avg Latency: 7576 usec

The Arduino standard SD.h library is based on a very old version of SdFat that I tried to keep small enough to run on 168 AVR processors.

After some patching to make the standard SD.h library work with 32 KB reads I get this result on Due with the above SD card:

Quote
Type is FAT32

File size 40MB
Buffer size 32768 bytes
Starting write test.  Please wait up to a minute
Write 229.87 KB/sec
Maximum latency: 805951 usec, Minimum Latency: 130704 usec, Avg Latency: 142545 usec

Starting read test.  Please wait up to a minute
Read 434.30 KB/sec
Maximum latency: 77666 usec, Minimum Latency: 72686 usec, Avg Latency: 75448 usec

Happiness is a factor of 16 in write speed and a factor of nearly 10 in read speed over the standard SD library.

I am running SPI at 42 MHz.  I built an SD shield with very short wire connecting the SD socket directly to the ISP connector.
836  Using Arduino / Storage / Re: Power consumption to write to SD card? on: November 25, 2012, 10:03:48 am
You won't get the answer with a DMM.  The max current occurs while flash is being programmed.  This is a brief interval and occurs at random times determined by the SD card's flash controller.  A DMM is too slow to measure this.

80 ma is common for consumer uSD cards.

Plan on 100 ma.  Some cards hit 150 ma peak for 10 us so good bypass caps are needed.

Some older SDHC cards require 200 ma for write.
837  Using Arduino / Storage / Re: How to work SD Library's "CardInfo" Program? on: November 24, 2012, 10:03:54 pm
A 168 is too small.
838  Development / Other Software Development / Re: RTOS candidates for Cortex M3/M4 duino boards. on: November 24, 2012, 09:55:01 pm
FreeRTOS can be used in commercial products http://www.freertos.org/a00114.html.  You must pay for the version called OpenRTOS.

There is also a free version of ChibiOS that can be used in commercial products http://www.chibios.org/dokuwiki/doku.php?id=chibios:license.  ChibiOS also has a non-free version.

I have looked at Contiki and at first glance it didn't look easy to port it as an Arduino library.  I haven't ruled it out and will study it more.
839  Using Arduino / Storage / Re: Second time i read dir arduino halts on: November 24, 2012, 09:21:53 pm
There are several problems.  The most serious is
Code:
String files [1][3];
may used with bad indicies here:
Code:
    files[j][i]=ename;
It looks like j could be one and i could be three.

Another problem is that the File wrapper for SdFat uses malloc so this assignment causes a memory leak unless you close entry. 
Code:
    entry =  root.openNextFile();
I didn't write the SD.h wrapper so you you nee to post this problem on the Arduino issues site.

There are other problems like you using the SD object and Sd2Card and SdVolume.  SD already has an instance of Sd2Card and SdVolume.

I stopped looking after I found these problems.
840  Products / Arduino Due / Re: Disappointing SPI performance with SdFat on: November 24, 2012, 03:30:07 pm
stimmer,

I understand the problem is with the library.  I want to use the standard SPI library since it's a pain to maintain my own SPI layer for every chip that runs SdFat.  Also, my SPI functions may not play well with the standard SPI library.

I would like to see optimized functions in the SPI library for sending and receiving blocks of data.

Pico,

I don't have a Maple version of SdFat with the new caching improvements for large memory chips.  Others have ported old versions of SdFat to Maple but these are slow.
Pages: 1 ... 54 55 [56] 57 58 ... 111