Go Down

Topic: sdfatlib: Possible to work around or minimize high-latency reads? (Read 921 times) previous topic - next topic

capsid

Hi! I've pieced together from a couple of forum threads that SD read speeds are not constant, with some reads taking much much longer for reasons I don't completely understand yet (something to do with FAT and block allocation?).  This plays out on my high-speed POV display when reading rows of bmp pixel data from an SD card.  Most reads (of a couple hundred bytes) take about 80-90 microseconds, but certain reads will take 1500 or even close to 3000 micros. This results in that row of the image being stretched and distorted.

Does anyone have any ideas how I can provide constant read speeds to the renderer? Having a full understanding of why some reads take longer could help.  The lib author explains in this post that some reads need to access FAT or other file structures which takes longer. http://forum.arduino.cc/index.php?topic=86880.msg651919#msg651919

Is it possible to create a special kind of file that doesn't need the structures which slow down reads so I could buffer image data into that file?  Would createContiguous() help here?

Any suggestions are greatly appreciated! :D

fat16lib

You can't avoid long read latencies.

SD cards are block devices with 512 byte blocks and an entire block must be read.  SdFat has an internal 512 byte cache so your low latency reads are just the time to copy data from the cache.  Longer latency reads fill the cache by reading a 512 byte block.

Your longest latency reads occur when SdFat must locate the next cluster.  This requires reading a 512 byte FAT block in addition to the data block.  If you SD card is properly formatted with 32 KB clusters, this will occur on 32 KB boundaries.

It is possible to avoid the FAT access by using raw reads of contiguous files.  The requires lots of memory and tricky programming.  You will still have latencies of at least 1000 usec when you read a block.  You may still see some 2000 usec or greater latencies depending on your SD card. 

pito

Quote
Does anyone have any ideas how I can provide constant read speeds to the renderer?

Define your data rates required..

fat16lib

You could use the same kind of buffering that audio players use.

I wrote a wave file player about five years ago that plays 16-bit 44.1 ksps wave files http://code.google.com/p/wavehc/downloads/list.  This player delivers data to an DAC at a constant rate of 88.2 KB/sec using a 328 Arduino.

This player uses a timer1 IRS to send data from buffers to the DAC.  Data is read from the SD into buffers in non-interrupt code.

Define your needs as pito suggested.

What is the data rate and how is the data sent to your POV display?  Give details of the interface to the display.

capsid

#4
Mar 20, 2014, 02:10 am Last Edit: Mar 20, 2014, 02:18 am by capsid Reason: 1
Hi, thanks so much for the help. I'll study your wave file player and see if I can figure it out.

As far as the data rates, my display is 50 pixels, 3 bytes per pixel, and when it looks good an entire cycle including drawing takes 1 ms, sooo... 150 bytes * 1000, 150KB/ sec? That seems impossible to reach. But if I can get 88.2KB/sec I'll gladly play with that.

I was considering buffering the images to a 128KB SPI SRAM and limiting the image to the SRAM size.  But I guess I should start with an interrupt based approach like you suggested and see how that goes.

Thanks again :D

capsid

Oh also I forgot to describe the interface to the display. Basically it's the FastLED library, so there is an array of 3 byte RGB elements. It reads pixel rows from the bitmap SdFile instance directly into the array. Then it calls FastLED.show() when it's time to write to the display.

fat16lib

Does your display interface use the SPI bus?

You can't use SPI in an ISR while a program like SdFat is doing a transfer in non-interrupt code.

fat16lib

#7
Mar 20, 2014, 01:34 pm Last Edit: Mar 20, 2014, 01:35 pm by fat16lib Reason: 1
I just looked at some tests I did for a similar project.  This project needed to read data from an SD and send data to a device over SPI at very high rates.

My solution was to read a contiguous file using the raw SD interface.  The standard SPI controller is dedicated to the SD.

With an industrial SD I could read the file at about 600 KB/sec with a maximum read latency of 1032 micros for a 512 byte block.
Quote

Starting read test
Read Done
File size: 51200 KB
Elapsed time: 81.950 seconds
Max read time: 1032 micros
Read rate: 624.773 KB/sec


I used an USART in SPI mode to send data to the device.  An AVR USART can run at 8 MHz in SPI mode.

I used a small RTOS, NilRTOS https://code.google.com/p/rtoslibs/, that I ported to Arduino.  This allows the thread that sends data to the device to run at high priority and the SD read thread to fill buffers at lower priority.

I did tests assuming a Mega or 1284P board would be used.  I am sure you could send 150 KB/sec with a Mega.   It might be possible to achieve this rate with an Uno but it would be marginal due to the limited 2 KB of SRAM.

Lower cost SD card can have occasional longer read latencies but would work with a Mega or 1284.

Here is a read result for a $5.00 4 GB SanDisk class 4 card.
Quote

Read Done
File size: 51200 KB
Elapsed time: 81.252 seconds
Max read time: 2488 micros
Read rate: 630.135 KB/sec

The 2488 micros would not be a problem with more buffering.

Go Up