journaling, of sorts

Hi there,

new-ish to the Arduino world, and just having some issues getting my head around how to do something.

In a Python script I have written, I have a class that sits in the background and sends data to a db ... if the machine should lose connectivity with the db server, the class will store the entries (in Memory and a file) until the connection is restored .... this journal file it keeps of things that have yet to make it to the db, so that if there is a power outage, it can come back and continue on where it left off.

I realize that there may not be a similar multitasking module for Arduino, and I am willing to forgo the journal file, and try and fake the rest of the functionality. So I can at least keep several sets of data on board till connectivity is resumed.

I know I could write to the flash on board, but I wanted to either keep that to an absolute minimum, or not at all. I know this will make it unsafe for power outages, but I am ok with that for now.

research so far, I have chosen an array, that I stick entries in, when the connectivity is down ... and will send them to the db when connectivity is restored.

Was wondering if this is a sensible way to approach it, or if there are better examples of dealing with connectivity, and still collecting data ?

cheers

Brett

We can task on Arduino just fine using event-driven code and NO special OS whatsoever. It's the lesson du jour every day here.

You can wire an SD module to Arduino cheaply. Just be sure to close your file and reopen it for append every so often if power outage can be a real problem, closing the file is how the FAT gets updated.

Many SD modules have just-get-by voltage leveling that over time ruins SD cards. It's about worth it to DIY an SD though it may be possible to either level the voltage between 5V Arduino and module yourself or use a 3.3V Arduino --- if the module will work on 3.3V, I never tried.
At 3.3V you can solder or clip to the card itself (or an SD adapter for microSD cards) and it will work.

You can hang a big EEPROM on SPI bus wires, you have a small EEPROM in the Arduino (Uno has 1K).

Whether it's sensible to store the data in SRAM really depends on how much data there is and how much SRAM you have available.

I did pick up 8K serial RAM chips cheap once. But they don't hold data through power-out like SD or EEPROM.

Look for the ESP8266 processor and boards such as NodeMCU or WeMOS D1 Mini. Sounds like the more appropriate solution for your problem. 3 MB on-board file storage, built-in WiFi connectivity, lots of RAM (some 56 kB) and high clock speed (80 or even 160 MHz). Costs about the same as a cheap Arduino.

wvmarle:
Look for the ESP8266 processor and boards such as NodeMCU or WeMOS D1 Mini. Sounds like the more appropriate solution for your problem. 3 MB on-board file storage, built-in WiFi connectivity, lots of RAM (some 56 kB) and high clock speed (80 or even 160 MHz). Costs about the same as a cheap Arduino.

Not to chase anyone away from Arduino, but since you already have code in Python, it’s worth noting that there exists a MicroPython environment that runs on ESP8266.

Both Arduino and Micropython on ESP8266 support a filesystem (in serial flash memory) where one could journal data, though this does raise the possibility of wear issues if logging is frequent. There’s also the potential of file corruption should power be lost while writing to a file.