Caching unsent data on SD card

I have a project that needs to write data to an SD card when internet connection is not available and then when internet is available it reads data back and send them out. This process is ongoing. The file will be opened for reading and writing. Is this possible with SD library? If I want to delete data I read from the cache file after I send them, how do I go about doing so? All I can think of is to start a new file and only copy the latter portion of the original cache file that has not been sent via internet. Then delete the original cache and rename the new with the original's name. I could create one file for every unsent data set but that will make too many files. The system sends data every minute and could lose internet connection for hours to days at a time. Thanks!

I have a similar intention. Just today my system lost its internet connection for an hour. My system uses an SD card to log all of its operations but it would be good to cache emails on an SD card if the internet connection drops.

You may be forced to write data to an SD file and append additional records as required. When the internet link is restored (assuming you can reliably detect that) you can then close the file and re-open it in read mode and read and send all the data.

Deleting data from a single file implies some sort of database and individual record tracking. That may be rather complex. But there is no reason not to achieve the end result using multiple sequentially numbered files so that you write each new record to the next sequential file. You then read the files sequentially and delete them after transmission.

But I see you think large numbers of records may need to be cached. Perhaps an intermediate solution would be best - store up to one hour's records in a single file and then move on to the next file.

I assume you know exactly how to share the SPI connectivity between ethernet and an SD card - if that is applicable to your hardware. It has certainly gaven me a lot of grief over recent weeks.

I hope this helps.

Catweazle NZ

I would recommend using a 'packet' system. This would allow one, or few large files.

Wrap each log entry and in the header include a length and time stamp or sequential ID, then simply append the packet to a file.
When the net is active, you can process the file from either end as the header could be a footer; this is good for streaming data of an unknown length, as the length is known once the stream stops.

If the net access is erratic, the time stamp or ID will not only allow the process to be restarted from the last good entry, but the server can ensure no duplicates, and possibly receive out of order data.

The length provides basic error QOS, also a CRC may be worth adding in.

Maybe starting a new file every time the net goes down, and deleting once processed.

CatweazleNZ,

I am using Arduino official Wifi shield so the library takes care of the CS pins on both the Wifi chip and SD card.

pYro_65,

I am thinking either having wrappers around data or having a separate index file with locations and lengths of each data set, unwrapped but delimited (possibly '\n'). I will enable the CRC on the sdfat library (I think I left them turned on after a stab at a photo logger project). The benefit of having one file is apparent. Everything is together. But if one has two files, then the index file can be easily rewritten, deleting all the entries that have been sent. I have to check for each data set to make sure they are sent before I remove their index from the index file. Then when the index file is empty, I remove the actual data file.

I got a PM about this project. So it's time to put a period on the post.

Please don't reply to this post.

I eventually moved to fatlib library. I was able to easily handle multiple files. I stored unsent data as packets in a cache.txt with length in the beginning of each packet. I used an index file to record the current position in the cache file. All packets after the index have NOT been uploaded, while all packets before it has been uploaded. This can prevent problems in case of loss of connection while the system tries to send cached data. Then when the last record of in the cache file is sent, both index and cache files are removed.

I can't post the code. It's intertwined with the rest of my code, which is several thousand lines and it's a contracted project.

In the future if I have time, I will make a library to buffer and resend so this function can be added to existing projects unlike this one.