Suggestings needed: open source data logger software dev.

I have the whole summer to develop something useful so I decide to work on an open source data logger software project to go with my hardware project. This logger will be able to log a variety of sensors with a number of logging schemes. Right now the idea is just in my head. I'd like some inputs from the community so I am on the right track to create an open source program that others can readily use.

Here are what I need some inputs on: 1. sensors. I have a number of them in mind for various situations. Please recommend other sensors and what someone might need to log them. Resistive sensors: thermistors, flex sensors, "pressure" sensors, all using simple voltage dividers Voltage sensors: LM35 etc temperature probes that output analog voltages, other sensors such as Vernier's analog probes including force, conductivity, temperature, pressure etc. Simple digital sensors: digital on/off switches for counting wind speed, water flow, liquid drop, counting interrupts of IR beam etc. Digital sensors with some simple protocols:sonic rangers with pulse width output Digital sensors with more advanced protocols: one-wire temperature probes, SDI-12 protocol temperature, dielectric constant sensors, serial port sonic rangers (free-running mode or triggered mode), other serial port sensors, I2C sensors, maybe SPI sensors but I don't know any of them.

  1. Logging schemes: I'll have triggered mode and automatic mode.

In trigger mode, logging occurs only when a trigger is sensed (pin change, analog voltage value etc.).

In automatic mode, logging occurs automatically driven by timers.

Under automatic mode: Continuous mode:

  • Short continuous mode logs data every xxx ms up to say 1 minute interval, driven by internal timers.
  • Long continuous mode logs data every xxx second, minute, hour, or day, driven by RTC.

Scheduled mode: Logging say every hour, day, week at a particular time

Object-wise, I will have:

Loggers: stores features of a logger, say input is A0, reading needs to be converted into temperature with what formula, output should have what unit and formatting when saving to SD card. The logging process will hopefully be finite state machines that make use of pin-change interrupt to not stall other logging activities.

Schedulers: stores running mode and affected loggers. Say a scheduler1 could have continuous mode at 1 per second and it will run logger1 and logger2 every second.

Triggers: This specifies the type of triggers. A scheduler will link to a trigger if the scheduler is in trigger mode.

I will make a text file syntax to spell out the settings. So the user can type up a setting file that has a number of loggers and schedulers, pop the SD card with the file into the logger hardware, and select the file on the logger hardware, the logger will run everything.

Suggestions, comments?

Hi Liudr,

I've been chugging away on an open source data logger system for a while now, and you can see the readers-digest version of it on my hackaday project listing. My focus so far has been on underwater applications, as that dovetails with my other interests:

http://hackaday.io/project/6961-the-cave-pearl-project

With a fairly recent drip sensor code build at:

http://github.com/EKMallon/The_Cave_Pearl_Project_CODEbuilds

Even though I'm still quite new to programming, I have come up with a few things that you might find interesting (or save you some time?)

I've used Pstring to dump data as asci in the eeprombuffer variable, which means that if I standardized my input variable names a little more I could essentially swap sensors without having to change any of the code handling the buffering / SD writes, etc. I currently have the code set to auto change my variable defs to match the sensors I have attached (with a ton of #ifdef & #endif statements) but it would not be hard to just make them all into "Generic integer input 1", "Generic integer input 2" etc, to make things more generic.

Also, I have sorted out how to have sensor interrupts waking the logger AND having regular timed alarms from the DS3231 SIMULTANEOUSLY in my drip sensors using pins 2&3, where drip impacts wakes the logger to update a counter, while the RTC alarm triggers the whole Timestamp/eeprom buffering/SD write cycles.

It took me a while to figure out how to nest the loops properly so that these different interrupts did not interfere with each other, and is probably the only thing that might interest someone with your level of experience.

Anyway, I look forward to seeing how your project develops, and hope I can learn from your work when you release your code.

cheers!