Online data logger with SD card backup

I'm trying to work out the best way to put this project together. Any guidance / tips would be appreciated.

I currently have a device built which detects when an input changes and uploads the value to a php file for processing and storage to a database. I want to expand on this and make it so that it logs the change locally and uploads it and so that if the connection is not available it will continue to log the states / values and upload once a connection is restored.

  • So something happens / state changes / value is periodically recorded.
  • Store value to SD card
  • Attempt to upload value
  • If upload successful add marker to value to indicate it has been uploaded
  • Check to see if any other values need uploaded and if so upload them and mark as uploaded

If somebody could help with breaking this down i think it'd help me. I've made a few projects interfacing to the web but not so much with SD card and RTC (required to timestamp the logged events).

If you need more information i'm happy to provide,
thanks in advance!

I would start with getting a data logging shield (adafruit.com has one) since those come with both an SD card slot and a RTC. It will also come with a library that will have lots of examples of how to read/write to the SD card.

As for the logic of your code, I would not store all the values on the SD card or else you will have to eventually deal with a full card. I would

  1. acquire the data
  2. Attempt to send data to the web
  3. if that fails, write data to SD card
  4. if it succeeds, check SD card for pending data and upload, then delete

You still face the possibility of losing the web connection long enough that your SD card could fill up.

Thanks for the advice, I've achieved most of what I want to already! I've decided to save everything to the SD log card and updata the line when it has been sucessfully uploaded. The arduino checks for any data that hasn't been uploaded periodically.