Using Lots of Data From SD Card

I'm trying to use an Arduino Uno to drive 200 LEDs and display animations. The animations are located on an SD memory card. There could be up to 250 animations on the SD card. Eventually, I will convert the sketch and project to use ONLY the ATmega328p chip instead of the Arduino (for cost reasons).

I understand the ATmega328p & Arduino Uno has the following memory:

  • Flash 32k bytes (of which .5k is used for the bootloader)
  • SRAM 2k bytes
  • EEPROM 1k byte

My sketch currently leaves about 700 Bytes of free SRAM, I could optimize this a little more.

So, I'm trying to figure out how I should architect this to quickly output the animations on the memory card. Considering one frame of an animation would take up 600-900 bytes, I certainly can't read the entire animation into SRAM.

Option 1: Load one frame into SRAM and then output it to the LEDs, one at a time.
But... How fast can I read 600-900 Bytes from an SD memory card using hardware SPI? Fast enough to keep the animation moving quickly?

Option 2: Stream the bytes for each pixel from the SD memory card and then output it to the LEDs, one LED at a time.
I think I've read that it would be faster to do Option #1 above? i.e. reading large chunks at a time instead of a couple bytes at a time.

Option 3: Load multiple frames into flash (as much as flash memory will allow), retrieve a frame from flash and then output it to the LEDs
But... I read that flash memory can only take 10,000 writes... I feel like I'll blow up my flash memory quickly this way.

Option 4: ??? Any other ideas? Maybe get a uC that has more memory than the ATmega328p??

Thanks in advance for the help!

ElectricDiode:
I'm trying to use an Arduino Uno to drive 200 LEDs and display animations. The animations are located on an SD memory card. There could be up to 250 animations on the SD card. Eventually, I will convert the sketch and project to use ONLY the ATmega328p chip instead of the Arduino (for cost reasons).

How do you plan on controlling the 200 LEDS?
How are the animations stored on the SD card?

unless you are mass producing, you will not save anything using atmega328 chip and building your circuit. you can get an arduino pro mini now for $3.45 and can use the full 32kb program space if you program via icsp.

anyway, if you are doing animation, is that at 15 frames per second? or probably slower right? at 15 frames, that gives you 66ms to setup the data and send it out.did you try this at all yet? sd read should take a shorter time that sd write. worse case you can use timer interrupt code to control your led line, reading from a circular buffer containing the frame data, and the sd card read will be in the main loop just continuously filling up the buffer.

SurferTim:

ElectricDiode:
I'm trying to use an Arduino Uno to drive 200 LEDs and display animations. The animations are located on an SD memory card. There could be up to 250 animations on the SD card. Eventually, I will convert the sketch and project to use ONLY the ATmega328p chip instead of the Arduino (for cost reasons).

How do you plan on controlling the 200 LEDS?
How are the animations stored on the SD card?

The uC clocks the data out to the LEDs. The animations are stored on the SD Card via simple bytes that represent each pixel's color. It's a file format that I created on a FAT32 SD Card.

unless you are mass producing, you will not save anything using atmega328 chip and building your circuit. you can get an arduino pro mini now for $3.45 and can use the full 32kb program space if you program via icsp.

Yes, I might mass produce this. I'd like to build it where it can easily scale. I don't think there's ever arduinos in something that is scalable. I'd like to create a finished product. I'm using the arduino right now to prototype. Am I missing something still?

anyway, if you are doing animation, is that at 15 frames per second? or probably slower right? at 15 frames, that gives you 66ms to setup the data and send it out.did you try this at all yet? sd read should take a shorter time that sd write. worse case you can use timer interrupt code to control your led line, reading from a circular buffer containing the frame data, and the sd card read will be in the main loop just continuously filling up the buffer.

The speed can vary from slow to as fast as the uC can run. I'm not exactly sure how many frames per second, but 15 frames per second would be a good target.
I haven't tried/tested this just yet. I'm building a prototype and at the moment I'm running my SD Card SPI at quarter speed because it doesn't work otherwise. I believe this is because I have about 8 inches of wire between the uC and the SD Card. That's why I wanted to ask this question because I didn't want to design everything with the idea that it would work, then come to find once I get the finished product put together it isn't fast enough.
I like your idea of having the interrupt downtime filling up the buffer. I suppose I'll work on building that and hope for the best!

If you are storing the data as XRGB_888, as in one byte per component / 3 bytes per pixel, you could try a different format.

XRGB_565 is what many TFT LCD displays use, it allows 65536 colours. Whats even better is a frame now fits in 400 bytes instead of 600, which is within the SD cards native block size. This would also greatly speed up SD card access times, and maybe allow multiple frames to be buffered at once.

pYro_65:
If you are storing the data as XRGB_888, as in one byte per component / 3 bytes per pixel, you could try a different format.

XRGB_565 is what many TFT LCD displays use, it allows 65536 colours. Whats even better is a frame now fits in 400 bytes instead of 600, which is within the SD cards native block size. This would also greatly speed up SD card access times, and maybe allow multiple frames to be buffered at once.

Yes, I am using 8 bits per color. XRGB_565 is an interesting idea. Although it would require conversion back to the XRGB_888, I might have to do that.

My experience is around 15KB/s continuous speed with SD card running at full SPI bus speed. You should get a different SD card setup to make it at least half speed SPI. Are you using resistors for level shifting or level shifter chips? The latter is faster. You will have a hard time "streaming" bytes from SD card since it is a block device. Things come in block at a time so pYro_65's idea to fit one frame within one SD card block is quite interesting. You may then pad each frame to the size of one SD card block and access will be faster.

How long is each animation, in terms of bytes?

liudr:
How long is each animation, in terms of bytes?

Each animation's frame is about 800 bytes. I use 3 Bytes for RGB Color + 1 Byte because I wanted to have some options available for a dynamic color option. So... 4 bytes X 200 LEDs = 800 Bytes per frame.

The animation can be unlimited number of frames.

liudr:
My experience is around 15KB/s continuous speed with SD card running at full SPI bus speed. You should get a different SD card setup to make it at least half speed SPI. Are you using resistors for level shifting or level shifter chips? The latter is faster.

I'm not sure what level shifting is... but perhaps my SD card breakout module has it? Here's the breakout module I'm using:

liudr:
You will have a hard time "streaming" bytes from SD card since it is a block device. Things come in block at a time so pYro_65's idea to fit one frame within one SD card block is quite interesting. You may then pad each frame to the size of one SD card block and access will be faster.

How exactly do I fit a frame in one SD card block? For example, if my animation doesn't start until byte 300 on the SD card, is it as simple as only reading 400 bytes at a times from that point on? Not sure if it has to fall exactly within bytes 0-400, 400-800, etc... on the SD card.

The data would have to be aligned with a disc sector.

I'm pretty sure each file is written on a 512 byte alignment. So make each frame <= 512 and place each in separate file.

Alternatively,
Pad out a frame to 512 bytes ( empty data at end ) and create either separate files or one large one.

With any of the three methods, there will be wasted SD space, but SD's usually have enough MB to deal with a bit of wastage.