Hi,
I'm working on building a machine learning algorithm to be used as a self contained library for use with an Uno or better. I'd like for the data gathered and learned to persist between poweroffs, but given that I'm implementing it with some basic statistics (mean, standard deviation, etc.), I need to keep arrays of a relatively large amount of previous readings (give or take 30 to 50 readings per sensor per situation), which can get very large very quickly. Specifically, way larger than I'd be able to store in EEPROM.
The problem I'm running into is that there doesn't seem to be a straightforward way to write to Flash during runtime. I know there has to be a way, or else the boatloader wouldn't work either, but all of the documentation and forum posts I've went through all talk about PROGMEM, which lets you put stuff in Flash during compile, but I need to be able to dynamically store and retrieve data whilst running.
My question is, how dirty am I going to have to get in order to get full read/write Flash access during runtime? Are there any simple workarounds? Or would I spend my time wiser coming up with a different algorithm that doesn't have to keep around such giant backlogs of data?
Put an SD Card Shield on the stack. That will give you almost unlimited memory.
You can also get I2C EEPROMS such as the LC256 with a lot more memory but the advantage of the SD card is that it's easy to pop it into your PC and do bulk updates.
Why not use external storage? FRAM over SPI has fast read/write access like SRAM, but is non-volatile, with >trillion write cycle capability (vs EEPROM 100,000).
"High-endurance 100 trillion (1014) read/writes"
How much storage do you need?
I've considered external storage, but like I said I'd really like for this to be as self contained as possible, to the point of being nothing more than an include for another sketch. Though I fear that with the size of the project so far, there wouldn't be a whole lot of room left for another sketch AND storage space, hah...
There isn't really a set amount of storage I need, I let the user choose a balance between precision and storage use.
At this point I think I might just end up developing a shield to go along with the library, definitely seems the way to go, because some quick calculations show me this can quickly go into the hundreds of kilobytes of data, which is obviously far beyond the scope of what a single Arduino could handle.
Thanks for the suggestions, I'll probably look into developing a shield using FRAM.
rutgersemp:
Thanks for the suggestions, I'll probably look into developing a shield using FRAM.
Personally, if I were developing this library, I wouldn't build a shield to go along with it. Instead, I would do as already suggested, and store the data on an SD card.
Doing this will open up the library to many more uses, including projects that may already have an Arduino embedded - but no room or pins for another shield.
There also wouldn't have to be an on-going effort to ensure compatibility between the shield and future Arduino board designs.
You would eliminate the need to support hardware (you're already going to be the main contact for software support on the library - why add to the potential pain?).
Finally, shields already exist for SD card interfacing; some of these shields contain other functionality as well (like MP3 player hardware, GPS, GSM, etc). There are also Arduino board designs out there with integrated SD card support. Basically - you get the hardware needed for free.
While it would be nice for everything to be self-contained, it probably isn't easily doable, and the room would be limited, as you have found out. Hobbyists can easily purchase an SD card interface or shield for their Arduino if needed, or they can implement an SD card interface in some other manner (seriously, you can easily make one using a dual-row pin header). SD cards are cheap, and easily available, and the interface library is mature and fairly robust.
It's a win all the way around, IMHO.
Calculators calculate the mean and SD for many readings, but they don't save all the data. You just need the number of samples, the sum of the samples and the sum of the squares and you can get mean and standard deviation.