Read variables from SD Card

Hi Folks,

I am trying to save variables on the SD card and read them again at power up. For example, say i am logging the Ah on a battery for capacity calculation, that is stored every so often.

After the arduino is powered off I would like to initialize such variable with the last recorded value, instead of 0.

Any ideas?

Sounds feasible. You would need to write the data to a file to save it, and read back the contents of the file at startup (with logic to get a sensible default if the data isn't present / can't be read). It would be easier to test and check correct behaviour if you write the data to the file using a textual format rather than a binary format.

If you don't have much data to save and don't need to save it very often, you could also save it in the EEPROM. There is an EEPROM Write Anything example in the playground which gives a couple of helper functions to make this simple to implement.

Care to post an example?

I currently save the data for logging purposes on the SD Card. What I don't know is how t read it back and assign it to a given variable.
This could also be useful to change parameters without reprogramming the device.

Why use an SD card for that? That's what EEPROM is for.

It Would take a long time to "Use Up" or wear out the Eeprom... and it does require less additional hardware..

Doc

PaulS:
Why use an SD card for that? That's what EEPROM is for.

Docedison:
It Would take a long time to "Use Up" or wear out the Eeprom... and it does require less additional hardware..
Doc

On this instance I plan to use a SD Card and not an EEPROM, that was the question on the topic.
Regards

On this instance I plan to use a SD Card and not an EEPROM, that was the question on the topic.

So, what, exactly, is the problem? Saving the contents of a variable to the SD card is simple. Reading the card is an order of magnitude harder, but 10 * 0 is still a small number.

So, what, exactly, is the problem? Saving the contents of a variable to the SD card is simple. Reading the card is an order of magnitude harder, but 10 * 0 is still a small number.

Catfish alert! :wink:

PaulS:

On this instance I plan to use a SD Card and not an EEPROM, that was the question on the topic.

So, what, exactly, is the problem? Saving the contents of a variable to the SD card is simple. Reading the card is an order of magnitude harder, but 10 * 0 is still a small number.

Care to post an example or some references?

This would be a bit easier if we knew exactly how you are saving records to the SD card to know how to parse the file. So, without that knowledge, I'll just have to speak on general terms.

Since you are writing to a SD card I'll assume you already know the reference page for the SD Library. Let me highlight some functions that you will want to read up on: [

[i]file.[/i]size()

](SD - Arduino Reference), [

[i]file.[/i]seek(pos)

](SD - Arduino Reference), and [

[i]file.[/i]peek()

](SD - Arduino Reference).

When you open a file object one of it's properties is a read/write pointer (or can be called a cursor). Some programming languages use two pointers, a read pointer and a write pointer. With the SD library we only have one pointer that does both jobs (I'm not totally clear if the reason is because of how the SD library was written, or if that is how C/C++ deals with file classes, but that level of detail really doesn't matter.) So one needs to remember to move the pointer to the end of the file before writing to avoid overwriting parts of the file.

When the last time your Arduino was on and writing to the data file it may have recorded lots of records. One should expect to find 0 to [very large number] of records. Think of the records as pages in a book. If you just want to know what is on the last page of the book do you start reading from the beginning, or do you turn to the last page? Reading the whole thing might be feasible if the "book" was a short 4-page menu, but what if it is the Oxford English Dictionary? Would you really read through all 12 volumes (for OED1) just to look at the last page? Unfortunately, there the analogy starts to break apart. One can't simply turn to the last saved record in the file. But one can turn to the last byte in the file using [i]file.[/i]seek(pos). One knows where that last byte is by using [i]file.[/i]size(), and one can see what that last byte is without moving the read/write pointer by using [i]file.[/i]peek(). Then it is a simple exercise of scanning backwards from the end of the file looking for your record separator (if you used [i]file.[/i]println() when writing to the file your separator would be , but since you can only scan one byte at a time, just scan for the character.) Then read and parse the end of the file much like one would read and parse the same data coming in over the serial port (except instead of using Serial.read() you would use [i]file.[/i]read()).

Hmmm... I suppose one could modify the analogy to fit the exercise better... Think about what you would do if you wanted to read the last chapter of Harry Potter and the Order of the Phoenix? Do you start flipping through pages from the front of the book, or start flipping pages backwards from the last page (page 870 of the US printing)? (I'm ignoring the possibility of a table of contents...)

Remember to put error corrections in there. Like consider what to do if the file isn't found, if the SD card isn't found, what to do if the last record isn't complete or is corrupted, etc.

My suggestion would be to write it as a function that you only call from setup(). That would keep the clutter down in setup(), and allow using local variables in the function that will cease to exist (and thus stop consuming valuable RAM) once the function completes. The 4 bytes of the unsigned long that you will be using to keep tabs of and manipulate the read/write counter should be a local variable, I'm sure there will be others...

Hopefully this is enough for you to cobble together some code that you can then post for us to critique. Not all of us are very tolerant of massive amounts of hand-waving. :wink:

Sembazuru:
Hopefully this is enough for you to cobble together some code that you can then post for us to critique. Not all of us are very tolerant of massive amounts of hand-waving. :wink:

Thats quite compreensive :wink:
At the moment I am just recording a few words every 5 seconds to a fixed .txt file. File is open, written and then closed. The values are later used for data logguing.

As you sugest i could start looking at the end of the file and look for the /CR line Each variable has its own file for the time being, however I am trying to use the TextFfinder library to read a single line, with individual values separated by comas, and load them into an array. This would only take place at system startup.

I am a bit confused on how I do add error checking capabilities, or for example, keep a backup should the file become corrupted?

casemod:
For example, say i am logging the Ah on a battery for capacity calculation, that is stored every so often.

Maybe you should give more concrete details about your solution - it may be possible to design away the requirements altogether.

For example, if you're trying to log how much charge has been removed from a battery you could record this as a sequence of deltas, so you don't ever need to hold a running total in memory.

This is for an EV (Electric vehicle) conversion.
I am creating my own ECU to drive the instrument cluster and other hardware that is now redundant.

Variables to be recorded are only the battery capacity and milleage.

PeterH:

casemod:
For example, say i am logging the Ah on a battery for capacity calculation, that is stored every so often.

For example, if you're trying to log how much charge has been removed from a battery you could record this as a sequence of deltas, so you don't ever need to hold a running total in memory.

I am curious about that. Will i be able to use those for data logguing purposes later?
What i need to know in total is:

Battery capacity (Charge discharge)
Charge - AC Charge (indicated with a digital I/O) and Regeneration (%)
Battery life cycle (basically (TOTAL AMPS IN/CAPACITY)
Cell mismatch (Difference between the two lithium packs, used to detect a bad cell)

Regards

casemod:
I am a bit confused on how I do add error checking capabilities, or for example, keep a backup should the file become corrupted?

No need for a backup for what I was talking about. What I was envisioning is because you (should) know what the data you are reading from the file should look like (because you wrote the code that actually did write it). If what you actually read from the file doesn't match the formatting of what you expect, presume there is an issue and either look for a previous record or just use a default value.

Sembazuru:

casemod:
I am a bit confused on how I do add error checking capabilities, or for example, keep a backup should the file become corrupted?

No need for a backup for what I was talking about. What I was envisioning is because you (should) know what the data you are reading from the file should look like (because you wrote the code that actually did write it). If what you actually read from the file doesn't match the formatting of what you expect, presume there is an issue and either look for a previous record or just use a default value.

I see. I am using something similar reading from the serial port. It checks if the data is a numeric constant, if not it discards and waits for the next result. I guess I could use something similar here.

I am just wondering what would happen if the actual log file got corrupted, for example a power down in the midle of a reading?

casemod:

Sembazuru:

casemod:
I am a bit confused on how I do add error checking capabilities, or for example, keep a backup should the file become corrupted?

No need for a backup for what I was talking about. What I was envisioning is because you (should) know what the data you are reading from the file should look like (because you wrote the code that actually did write it). If what you actually read from the file doesn't match the formatting of what you expect, presume there is an issue and either look for a previous record or just use a default value.

I see. I am using something similar reading from the serial port. It checks if the data is a numeric constant, if not it discards and waits for the next result. I guess I could use something similar here.

I am just wondering what would happen if the actual log file got corrupted, for example a power down in the midle of a reading?

Honestly there is only so much you can plan for. There is always the slight possibility that if power was interrupted during a write to the SD card you could actually corrupt the SD card itself. The power would have to be lost at just the right (or wrong?) time. Best case would be a partial record saved.

Depending on how critical not loosing data is, you might want to consider a backup battery for the Arduino. Sense the main input power to the Arduino (I haven't thought through far enough to flesh out an actual circuit yet), and when the main power dies, prepare for a clean power off (make sure any records that need to be written are completely written) and then switch off the power from the backup. I personally think that this is overkill, but you may have different priorities.