Pages: [1] 2   Go Down
Author Topic: Read SD card for data  (Read 1144 times)
0 Members and 1 Guest are viewing this topic.
Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm trying to figure out how I can accomplish reading an SD card without it blocking the running loop.  Basically what I'm thinking of doing is storing POV patterns on an SD card, then have running code read in the first pattern and push it out to the LED string.  When a certain time has passed, I need it to go back to the SD card and read the next pattern and display it on the LED string.

The problem here is that I don't want the POV display to stop while the code reads in the next file from the SD card.  It does need to read in the entire contents of the file before being able to process and display it (and yes, I know I'm limited to the amount of information I can cram in memory.)

So is there a way to do this?  Or should I consider perhaps adding a fast external flash memory module and fill that in with the contents of the SD card upon start up and only use that for the running code?
« Last Edit: June 02, 2013, 06:27:06 pm by KirAsh4 » Logged

Sydney, Australia
Offline Offline
Edison Member
*
Karma: 27
Posts: 1187
Big things come in large packages
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Have you done any tests to see if it is possible to read the amount of data in the time you are expecting?

A simple loop with timing values (I guess microsecond rather than millis) either side of the 'read' will give you some base data to look at to decide feasibility of the approach.
Logged

Arduino libraries http://arduinocode.codeplex.com
Parola hardware & library http://parola.codeplex.com

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 551
Posts: 46244
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
The problem here is that I don't want the POV display to stop while the code reads in the next file from the SD card.
How is the POV display happening?

Reading from an SD card, especially when it is necessary to locate and open the file, will take time. It is unrealistic to expect that to not interfere with POV code.
Logged

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How is the POV display happening?

Right now, as a constant running loop:
Code:
void loop() {
  uint16_t reps, c, px;

  #ifdef PAT1
  for (reps = 0; reps < REP1; reps++) {
    for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {
      for (px = 0; px < NUM_LEDS; px++) {
        unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));
        unsigned char r = cur & Rmask;
        unsigned char g = (cur & Gmask) << 2;
        unsigned char b = (cur & Bmask) << 4;
        leds1[px] = CRGB(r, g, b);
      }
      LEDS.show();
      _delay_us(ELONG1);
    }
    _delay_ms(PAUSE1);
  }
  #endif

  #ifdef PAT2
  ...
  #endif

  // etc., etc.
}

Eventually there will be other pieces added (such as a battery voltage check) so things will change just a tad.  Somewhere in there I'm going to be reading the SD card to get those patterns.  What I want to do is add patterns as individual files on the SD.  When the code starts up, it will collect all the file names from the card and store that list, then cycle through them one by one by opening each file and reading its contents before passing it to the above loop to display (that will do away with the #ifdef #endif lines.)
Logged

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Ok so I'm stuck, and since I stared this thread here about reading the SD, I will keep this here as well, even though it could also go in the Programming section.  So, right now, the patterns are stored in PROGMEM like so:
Code:
#define PAT1
prog_uchar img1[] PROGMEM = {
0x52,0x52,0x52,0x52,0x54, ...,0x7E,0x7E,  (48 values per line)
0x72,0x74,0x74,0x4A,0x54, ...,0x7E,0x7E,
etc., etc.
};

#define REP1 172
#define PAUSE1 0
#define ELONG1 1411

Each pattern is different of course, so the amount of lines of color codes varies for each.  All of them end with the same three lines at the bottom (with different values.)  Then the code snippet I posted above will read that data in and pass it on to the LED string.

What I'd like to do is take that and put it in a file on the SD card; each pattern in a separate file.  Right now, depending on the complexity of the patterns, I can have somewhere between 2 to 15 patterns within the 32K on an Uno.

What I'd much rather do is offload all the patterns to an SD card, then have the running code read them in as it needs them.  Each pattern gets displayed for a specific amount of time, some for 5 seconds, others for 15 seconds, others for much longer (that's where the REP* value comes into play.)

This is where I'm stuck: how do I store the data offline in a file, in a specific structure, then read it back in for the program to use.  I don't think I need to use PROGMEM with this, so I'll just be storing the actual data and three variables at the end.  But how do I read it back in and how do I store it temporarily for the program loop to use?  I say temporarily because when it needs to go to the next pattern, it can erase the old pattern and read in a new one.

And then there is the issue of the delay in accessing and reading the data that I want to figure out how to minimize.  I don't want the program to stop displaying a pattern while it accesses the SD card and reads in data.  I don't know if that can even be done.  Maybe a second AVR that reads in data then waits for the primary one to request a new pattern and push it through a serial connection between the two?  I don't know ...

Ideas, suggestions?
Logged

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Have you done any tests to see if it is possible to read the amount of data in the time you are expecting?

A simple loop with timing values (I guess microsecond rather than millis) either side of the 'read' will give you some base data to look at to decide feasibility of the approach.

Here you go:
Code:
Files on card:
PATTERN1.TXT   2013-06-04 04:13:56 18764
PATTERN2.TXT   2013-06-04 04:14:20 16586
PATTERN3.TXT   2013-06-04 04:14:48 12955

Elapsed read-in time for PATTERN1.TXT: 717ms
Elapsed read-in time for PATTERN2.TXT: 635ms
Elapsed read-in time for PATTERN3.TXT: 495ms

The timer starts the moment it opens the file and stops when it closes the file.  The POV display updates a lot faster than that (its pauses are in microseconds.)  So I need to figure out a different approach to this, one where reading the file on the SD card isn't going to interfere with the POV display.  I'm not sure I can do that on a single controller, but I could be wrong.  Perhaps with some serious time management where I monitor the elapsed time and if it's time for the POV to update, do that, then go back to reading the SD card ... I don't know.
Logged

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

Code:
for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {

is sizeof a macro, or is this (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)) being calculated every loop? ie does the compiler constant it out?

I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

Code:
unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.

Also: if you are going to generate text files, is there any advantage in doing all your calculations up front, storing pre-calculated RGB values for each pixel and read them in to be assigned to the LED sans manipulation? When you generate the text file you could specify the number of leds in use.

Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Another idea I had is to have a very simple "transition" POV image that can run with minimal processing effort between SD-retrieved images.
Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

Code:
unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.

Further to this - on the SD-retrieve image transition, you could read in the chars one at a time in this loop (assuming char read is possible and fast enough) and store it internally the first time reading the file in. You'd get some sort of merge (?) of the current and incoming POV but that might even look cool(ish).

Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

For that test, the "loop" was literally
Code:
 t = millis();

  if (!myFile.open("pattern1.txt", O_READ)) {
    Serial.println("Can't open file!");
    return;
  } else {
    int data;
    while ((data = myFile.read()) >= 0) {}
    myFile.close();
  }
  t = millis() - t;

  cout << pstr("Elapsed read-in time for PATTERN2.TXT: ") << t << "ms" << endl;

So it wasn't doing anything with the data, just reading line for line till it reached the end of the file.

Code:
for (c = 0; c < (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)); c++) {

is sizeof a macro, or is this (sizeof(img1) / (sizeof(prog_uchar) * NUM_LEDS)) being calculated every loop? ie does the compiler constant it out?
No it doesn't.  sizeof

I have no experience with SD card reading (yet!!) so forgive me if this is misguided, but could you simply open the file, and then replace

Code:
unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

with an equivalent that reads the byte from the SD card open file handle instead? Rather than reading the entire file in at once.
The thing is, doing *anything* with ah SD is always going to be slower than doing anything else on the MCU.  The POV display changes very fast, in the microseconds.  Reading an SD card will take milliseconds.  So the time it takes to read something from SD will be larger than the time it takes for the POV to update.  That will cause problems.  Whether I read one single byte, or an entire line, it will block everything else.  Though I am working to come up with a different solution ...

Also: if you are going to generate text files, is there any advantage in doing all your calculations up front, storing pre-calculated RGB values for each pixel and read them in to be assigned to the LED sans manipulation? When you generate the text file you could specify the number of leds in use.

One idea is to store the data in binary format.  This will cut down on the time it takes to manipulate the data to get the r, g, and b values.  But, not being a C/C++ programmer, it's a (very) slow process for me ...  Little by little.
Logged

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How quick is that loop? Have you benchmarked it to see where the bottlenecks are (if any?)

For that test, the "loop" was literally

Nope, I meant your POV loop. If you speed that up, you have more idle time that could be used reading the SD card.

I also just did some math.

This file
PATTERN1.TXT   2013-06-04 04:13:56 18764

was read in its entirety in 717ms.

That's worst case scenario of 26 characters / ms. Because it includes a (typically) slow file open and close operation.

This means reading in 1 character (worst case scenario) only takes 38 microseconds.

If your memory access line

Code:
unsigned char cur = (uint32_t)(pgm_read_byte_near(&img1[px + c * NUM_LEDS]));

takes 38 uS or longer, you can do a direct replacement and would not notice a difference.

Also, you pause after each LED set up:

Code:
LEDS.show();
_delay_us(ELONG1);

In your cut down example that's a 1411 uS or 1.411ms pause.

You could read in 1141/38 = 30 characters during that pause.

Assuming you are reading 48 values at a time (based on your line length mentioned in comments), you only need to read in an additional 18 values for each loop.

Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If you  have the memory, one way of speeding up code is to unroll loops. Not sure if that works on this architecture.

Remove static calculations to variable and compute the result before the loop (unless the compiler does this for you - would have to check the er bin or hex or whatever).

Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yeah, in theory that would work, however the ELONG pause is different for each image.  The images aren't all the same exact width, so the pause for each refresh is different.  So for each image I'll have to figure out how much time I have to do an SD read and shove into an array.

I'm not opposed to time slicing the reads.  However, the ability to do that ...
Logged

Melbourne, Australia
Offline Offline
God Member
*****
Karma: 8
Posts: 567
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sure. If the image is narrower, the pause is shorter, but then the number of characters being read per line is less also, yeah?
Logged

Windows serial port monitor: Tellurium | Arduino serial port debugging library: DBG | Cusom LCD char generator | Technical questions will only be answered in forum threads

Colorado
Offline Offline
Edison Member
*
Karma: 47
Posts: 1562
Reviving dead brain cells with Arduinos.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

No, the amount of values per line remains the same.  It's the amount of lines that change.  Images are 48 pixels high.  The data (in PROGMEM) is stored one full LED string at a time.  So take the image and rotate it 90 degrees clockwise.  Now you have lines that are 48 pixels long, representing the 48 pixels.  The loop will fetch an entire line from memory and push it out to the LED string which then updates.  Then it cycles to the next line.
Logged

Pages: [1] 2   Go Up
Jump to: