Read SD card for data

I'm trying to figure out how I can accomplish reading an SD card without it blocking the running loop. Basically what I'm thinking of doing is storing POV patterns on an SD card, then have running code read in the first pattern and push it out to the LED string. When a certain time has passed, I need it to go back to the SD card and read the next pattern and display it on the LED string.

The problem here is that I don't want the POV display to stop while the code reads in the next file from the SD card. It does need to read in the entire contents of the file before being able to process and display it (and yes, I know I'm limited to the amount of information I can cram in memory.)

So is there a way to do this? Or should I consider perhaps adding a fast external flash memory module and fill that in with the contents of the SD card upon start up and only use that for the running code?

Have you done any tests to see if it is possible to read the amount of data in the time you are expecting?

A simple loop with timing values (I guess microsecond rather than millis) either side of the 'read' will give you some base data to look at to decide feasibility of the approach.

The problem here is that I don't want the POV display to stop while the code reads in the next file from the SD card.

How is the POV display happening?

Reading from an SD card, especially when it is necessary to locate and open the file, will take time. It is unrealistic to expect that to not interfere with POV code.

PaulS:
How is the POV display happening?

Right now, as a constant running loop:

void loop() {
  uint16_t reps, c, px;

  #ifdef PAT1
  for (reps = 0; reps < REP1; reps++) {
    for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {
      for (px = 0; px < NUM_LEDS; px++) {
        unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));
        unsigned char r = cur & Rmask;
        unsigned char g = (cur & Gmask) << 2;
        unsigned char b = (cur & Bmask) << 4;
        leds1[px] = CRGB(r, g, b);
      }
      LEDS.show();
      _delay_us(ELONG1);
    }
    _delay_ms(PAUSE1);
  }
  #endif

  #ifdef PAT2
  ...
  #endif

  // etc., etc.
}

Eventually there will be other pieces added (such as a battery voltage check) so things will change just a tad. Somewhere in there I'm going to be reading the SD card to get those patterns. What I want to do is add patterns as individual files on the SD. When the code starts up, it will collect all the file names from the card and store that list, then cycle through them one by one by opening each file and reading its contents before passing it to the above loop to display (that will do away with the #ifdef #endif lines.)

Ok so I'm stuck, and since I stared this thread here about reading the SD, I will keep this here as well, even though it could also go in the Programming section. So, right now, the patterns are stored in PROGMEM like so:

#define PAT1
prog_uchar img1[] PROGMEM = {
0x52,0x52,0x52,0x52,0x54, ...,0x7E,0x7E,  (48 values per line)
0x72,0x74,0x74,0x4A,0x54, ...,0x7E,0x7E,
etc., etc.
};

#define REP1 172
#define PAUSE1 0
#define ELONG1 1411

Each pattern is different of course, so the amount of lines of color codes varies for each. All of them end with the same three lines at the bottom (with different values.) Then the code snippet I posted above will read that data in and pass it on to the LED string.

What I'd like to do is take that and put it in a file on the SD card; each pattern in a separate file. Right now, depending on the complexity of the patterns, I can have somewhere between 2 to 15 patterns within the 32K on an Uno.

What I'd much rather do is offload all the patterns to an SD card, then have the running code read them in as it needs them. Each pattern gets displayed for a specific amount of time, some for 5 seconds, others for 15 seconds, others for much longer (that's where the REP* value comes into play.)

This is where I'm stuck: how do I store the data offline in a file, in a specific structure, then read it back in for the program to use. I don't think I need to use PROGMEM with this, so I'll just be storing the actual data and three variables at the end. But how do I read it back in and how do I store it temporarily for the program loop to use? I say temporarily because when it needs to go to the next pattern, it can erase the old pattern and read in a new one.

And then there is the issue of the delay in accessing and reading the data that I want to figure out how to minimize. I don't want the program to stop displaying a pattern while it accesses the SD card and reads in data. I don't know if that can even be done. Maybe a second AVR that reads in data then waits for the primary one to request a new pattern and push it through a serial connection between the two? I don't know ...

Ideas, suggestions?

marco_c:
Have you done any tests to see if it is possible to read the amount of data in the time you are expecting?

A simple loop with timing values (I guess microsecond rather than millis) either side of the 'read' will give you some base data to look at to decide feasibility of the approach.

Here you go:

Files on card:
PATTERN1.TXT   2013-06-04 04:13:56 18764
PATTERN2.TXT   2013-06-04 04:14:20 16586
PATTERN3.TXT   2013-06-04 04:14:48 12955

Elapsed read-in time for PATTERN1.TXT: 717ms
Elapsed read-in time for PATTERN2.TXT: 635ms
Elapsed read-in time for PATTERN3.TXT: 495ms

The timer starts the moment it opens the file and stops when it closes the file. The POV display updates a lot faster than that (its pauses are in microseconds.) So I need to figure out a different approach to this, one where reading the file on the SD card isn't going to interfere with the POV display. I'm not sure I can do that on a single controller, but I could be wrong. Perhaps with some serious time management where I monitor the elapsed time and if it's time for the POV to update, do that, then go back to reading the SD card ... I don't know.

How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {

is sizeof a macro, or is this (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)) being calculated every loop? ie does the compiler constant it out?

I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.

Also: if you are going to generate text files, is there any advantage in doing all your calculations up front, storing pre-calculated RGB values for each pixel and read them in to be assigned to the LED sans manipulation? When you generate the text file you could specify the number of leds in use.

Another idea I had is to have a very simple "transition" POV image that can run with minimal processing effort between SD-retrieved images.

aarondc:
I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.

Further to this - on the SD-retrieve image transition, you could read in the chars one at a time in this loop (assuming char read is possible and fast enough) and store it internally the first time reading the file in. You'd get some sort of merge (?) of the current and incoming POV but that might even look cool(ish).

aarondc:
How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

For that test, the "loop" was literally

  t = millis();

  if (!myFile.open("pattern1.txt", O_READ)) {
    Serial.println("Can't open file!");
    return;
  } else {
    int data;
    while ((data = myFile.read()) >= 0) {}
    myFile.close();
  }
  t = millis() - t;

  cout << pstr("Elapsed read-in time for PATTERN2.TXT: ") << t << "ms" << endl;

So it wasn't doing anything with the data, just reading line for line till it reached the end of the file.

aarondc:

for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {

is sizeof a macro, or is this (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)) being calculated every loop? ie does the compiler constant it out?

No it doesn't. sizeof

aarondc:
I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.

The thing is, doing anything with ah SD is always going to be slower than doing anything else on the MCU. The POV display changes very fast, in the microseconds. Reading an SD card will take milliseconds. So the time it takes to read something from SD will be larger than the time it takes for the POV to update. That will cause problems. Whether I read one single byte, or an entire line, it will block everything else. Though I am working to come up with a different solution ...

aarondc:
Also: if you are going to generate text files, is there any advantage in doing all your calculations up front, storing pre-calculated RGB values for each pixel and read them in to be assigned to the LED sans manipulation? When you generate the text file you could specify the number of leds in use.

One idea is to store the data in binary format. This will cut down on the time it takes to manipulate the data to get the r, g, and b values. But, not being a C/C++ programmer, it's a (very) slow process for me ... Little by little.

KirAsh4:

aarondc:
How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

For that test, the "loop" was literally

Nope, I meant your POV loop. If you speed that up, you have more idle time that could be used reading the SD card.

I also just did some math.

This file
PATTERN1.TXT 2013-06-04 04:13:56 18764

was read in its entirety in 717ms.

That's worst case scenario of 26 characters / ms. Because it includes a (typically) slow file open and close operation.

This means reading in 1 character (worst case scenario) only takes 38 microseconds.

If your memory access line

unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

takes 38 uS or longer, you can do a direct replacement and would not notice a difference.

Also, you pause after each LED set up:

LEDS.show();
_delay_us(ELONG1);

In your cut down example that's a 1411 uS or 1.411ms pause.

You could read in 1141/38 = 30 characters during that pause.

Assuming you are reading 48 values at a time (based on your line length mentioned in comments), you only need to read in an additional 18 values for each loop.

If you have the memory, one way of speeding up code is to unroll loops. Not sure if that works on this architecture.

Remove static calculations to variable and compute the result before the loop (unless the compiler does this for you - would have to check the er bin or hex or whatever).

Yeah, in theory that would work, however the ELONG pause is different for each image. The images aren't all the same exact width, so the pause for each refresh is different. So for each image I'll have to figure out how much time I have to do an SD read and shove into an array.

I'm not opposed to time slicing the reads. However, the ability to do that ...

Sure. If the image is narrower, the pause is shorter, but then the number of characters being read per line is less also, yeah?

No, the amount of values per line remains the same. It's the amount of lines that change. Images are 48 pixels high. The data (in PROGMEM) is stored one full LED string at a time. So take the image and rotate it 90 degrees clockwise. Now you have lines that are 48 pixels long, representing the 48 pixels. The loop will fetch an entire line from memory and push it out to the LED string which then updates. Then it cycles to the next line.

And does the LEDS.show then use a multiplexer to address the individual leds?

I'm interested in the solution you come up with.

If it were me and the timing was too tight to slice the read up inside the loop I'd be considering a second processor

It's a library call. It simply pushes the data out via SPI to the LEDs (which have their own ICs.) That part is blazingly fast.

And with a second processor, you're still dealing with the delay in sending the data from one to the other. It's a large chunk of data that needs to be read into memory, the full image. I don't want to constantly be reading the SD, I just want to read the image in once, use it for whatever time frame, erase it from memory and start using the next one in line. How I get that next image, without having a visible interrupt between them, that is what's at stake here.

with 2 processors you could

(Assuming external flash memory is faster)

1. read the next image into external flash memory using a slave processor
2. read the next image into internal memory from flash memory using the master

OR

1. read the next image into P1 while displaying the current image with P2
2. switch the display processor from P2 to P1
3. read the next image into P2 while displaying the current image with P1

aarondc:
with 2 processors you could

(Assuming external flash memory is faster)

1. read the next image into external flash memory using a slave processor
2. read the next image into internal memory from flash memory using the master

OR

1. read the next image into P1 while displaying the current image with P2
2. switch the display processor from P2 to P1
3. read the next image into P2 while displaying the current image with P1

I really, really, really want to keep it down to a single processor.