Questions about interfacing with SD card

Other Hardware Storage
1

On a project I'm currently doing R&D work on, I'll be incorporating an SD card for reading of external data. What I'd like to do is store instructions, specifically in_idx, angle_pan, speed_pan, angle_tilt, speed_tilt, and a timestamp index. Imagine an Excel file with all the data in it. in_idx is simply the instruction index.

My question is, is it possible to "stream" this data. In other words, rather than having the MCU open the file and read the whole thing in memory to execute, is it possible to read-as-needed? What I'm thinking is, let's say the first few instruction look like this:

in_idx  angle_pan   speed_pan   angle_tilt   speed_tilt   timestamp
     0         90           2           90            2           5
     1         45           4           90            0           2
     2        135           4           90            0           5

This tells me from the first instruction, that both pan and tilt servos need to go to their 90 degree position at speed = 2. The '5' timestamps at the end means nothing happens for 5 seconds after that, so the MCU knows not to come back for another instruction for 5 seconds. (The board will have an RTC on it for accurate timings and syncing with other units nearby.)

After those 5 seconds have elapsed, it reads the next set which is 45, 4, 90, 2, 2: move the pan servo to 45 degrees at speed = 4, keep the tilt servo at 90, and wait 2 seconds before getting the next instruction.

After 2 seconds, read the next instruction which tells the pan servo to go to 135 degrees at speed = 4, keep the tilt servo at 90, and wait 5 seconds for the next instruction.

I suppose I can open/close the file each time, but I'm more concerned about delays in doing that. Basically I want a system I can time accurately to music. I can tear the audio apart and get exact timings for it. I want to be able to feed the timing data (modified of course) into a table and onto the SD card for the MCU to then read from.

What caveats do I need to look out for? What potential problems and/or headaches will I encounter here? I know I need to figure out how long to wait before expecting the servo to be where I sent it to and feeding it another instruction (otherwise it will never reach its intended target angle.)

Honestly, I don't know if this is the best way of doing this process, so I am open for suggestions here. The reason I want to incorporate an SD card for this is two fold,
a) I suspect the data will be rather large to fit it all in 32K of memory (actually, I know it won't fit.)
b) Using an SD card means I can set all the instructions in Excel, and export it as a comma delimited file and onto the SD card from my computer, then just pop the card onto the board.

Ideas/suggestions?

2

My question is, is it possible to "stream" this data. In other words, rather than having the MCU open the file and read the whole thing in memory to execute, is it possible to read-as-needed?

Not only is is possible, it is impossible not to. The function to read from the SD card, like the function to read from the serial port, reads one character at a time.

Behind the scenes, a larger portion of the file is read into a buffer. It it that buffer that SD.read() accesses, not the SD card itself. No, you can't change that behavior.

in_idx is simply the instruction index.

So, why bother? This value, if needed, can be inferred from the record number.

I suppose I can open/close the file each time, but I'm more concerned about delays in doing that.

You don't need to. Open the file once. Close the file once. Read the file in a while loop that executes until the whole file has been read.

What potential problems and/or headaches will I encounter here?

Anything from none at all to complete migraines. Depends on whether you test the code as you go, or try to write it all first, then test.

3

PaulS:
Behind the scenes, a larger portion of the file is read into a buffer. It it that buffer that SD.read() accesses, not the SD card itself. No, you can't change that behavior.

Well that's fine, I don't plan on changing it either. So if a large portion is read into the buffer, how does it know when to go back and read more? Or is that what I need to control?

PaulS:
So, why bother? This value, if needed, can be inferred from the record number.

Because I'm used to doing that for MySQL tables. :slight_smile: Each record has an index number, so I can read only the record that has that specific index number (or record within a specific range.) Whether I will be reading things sequentially, or jumping back and forth, won't matter as long as there's an index number. But, if it's not needed here since I will be reading sequentially at all times, then I won't bother adding that unnecessary data.

PaulS:
You don't need to. Open the file once. Close the file once. Read the file in a while loop that executes until the whole file has been read.

Sounds easy, perhaps easier said than done. There will be other stuff going on of course, such as controlling the actual servos and also communicating via Ethernet (though this is still being debated and may not make it into the final project, or at least not the first iteration.)

PaulS:
Anything from none at all to complete migraines. Depends on whether you test the code as you go, or try to write it all first, then test.

Gotcha.

Now, I just thought of something else, is there a way to make the SD card show up as a mass storage device when the unit is plugged into a computer's USB port? Let's me explain the controller board here. It has an AVR (with Arduino bootloader), an RTC w/ battery, FTDI IC, SPI breakout, and the card reader (with hex converter). A pluggable stack above it will have the servo connections which simply tie into the Arduino pins below. Same principle as any Arduino shield.

The FTDI USB connection is to download new sketches (because I just don't like using SPI for that every time, personal choice.) So I'm curious if the SD card can then shows up as a mass storage device on the computer when a USB cable is plugged in. This will save me from having to remove it each time I want to download a new instructions file to it.

4

So if a large portion is read into the buffer, how does it know when to go back and read more? Or is that what I need to control?

When you read(), as with Serial.read(), the character is removed from the buffer. When the buffer is empty, a read() causes another buffer-full of data to be fetched. This, of course, is time-consuming.

But, if it's not needed here since I will be reading sequentially at all times, then I won't bother adding that unnecessary data.

That will make reading the data faster, since there is less of it.

5

PaulS:

So if a large portion is read into the buffer, how does it know when to go back and read more? Or is that what I need to control?

When you read(), as with Serial.read(), the character is removed from the buffer. When the buffer is empty, a read() causes another buffer-full of data to be fetched. This, of course, is time-consuming.

So it might actually be worth trying to figure out when to time those reads? If I know the MCU is going to be idling for 5 seconds between instructions, maybe trigger a read then? Pondering ...

Any suggestions on the other part?

6

The data is read from the file in 512 byte chunks. You can't force a read. When the buffer that was last read is empty, another read from the file occurs. You can't control when that happens, unless you make each record (or collection of records) exactly 512 bytes.

No idea about HID or how you would make your device appear as a HID.