Arduino/Rpi sensor network log

Hey all,

After browsing through the forums, and reading some other stuff aswell, I decided to start keeping a log of my sensor network build. I’ve been ogling at the builds of stansleyseow, nathan chantrell (TinyTX) and maniacbugs for too long now, so time to get going and build what i want to have.

So what do I want? Or rather, how do I currently envision the end situation to be :wink: ? We all know that during projects like this, the ultimate goal tends to shift around a bit. But for the time being …

I want to have a sensor network that monitors temperature, air humidity and soil humidity (think indoor plants, garden). An update once every half hour or hour or so is fine. It needs to be low-power, wireless and more or less stable. Not every sensor needs to monitor everything. In the end, the soil humidity in the garden will be controlled by an arduino based, valve operated watering system with drip-hoses throughout the (small) garden. Wireless. Everything will in the end be controlled from a web interface running on a Raspberry Pi.
The Pi is also used to control my 433MHz based klik-aan-klik-uit or Elro/Intertechnik light switches and dimmers (but as that isn’t arduino based, I’ll keep it out of the forum).

In a nutshell what I envision, but where are we now?

I already had a raspberry Pi, that originally functioned as XBMC box, however I replaced it with a BR player with Plex. EVerything else in the house was doing plex, and Plex on Rpi sucked so badly I didn;t want to go ahead with that. And it was my birthday :wink:

I started out by getting the Raspi talking to the RF module of my choice, the nRF24L01+. Those things are tiny, very very cheap (how can you build 10 and send them to the Netherlands for $9.80 ?!?) and perform nice enough. 250kbit is ample bandwidth, range is good enough, especially with the versions that have an “amplifier”, but more importantly a normal antenna.
THat worked out ok, and I wrote a small C++ program that listens to the module, and, when data is available pipes it through to an external script (php, love it) for parsing/handling of the data. Data is stored in a MySQL db running on the FreeNas (which also runs my own domain).

And now, more importantly, the arduino-ish side of the story. I had the Uno. Clone. I had a bunch of cables connecting it to the RF24 module. THat sucked. So I created a “shield” for the nRF24L01+:

And the shield on the Uno:

That worked a lot better. I tested all 10 moduled I had, and every single one worked fine. nothing wrong with them. The claims that the range is a lot more than it actually is, I don;t care about that. They;re dirt cheap, and not too badly built.
This one is currently chatting away, telling the Rpi at least two times per minute what temperature it is in the study. So it works.

Enter the Attiny. I wanted low power and not too often having to change batteries. So, that’s what I’m working on now.
I created a second shield for programming the tiny with ArduinoISP (info on google and everywhere in the forums, not many people build the three leds on it though, they are (from left to right) heartbeat, error, comms, see ArduinoISP code comments):

And again, on the uno:

After some soldering, re-soldering, tweaking some connections and replacing the 10uF cap (bad one), I finally got it to work (took me half a day for god’s sake :slight_smile: ). The worst was that when I was testing the tiny I was programming, I continually kept holding the LED the wrong way around (I cut the leads to the same length, stupid me :* ).

Anyway, current stance is:

  • Raspi as a base, working (basics in place, listener, data parser, sql db, simple web page with one graph)
  • RF24’s working on the uno
  • Attiny programmer working
  • Most of the materials have arrived in the mean time

Now I need time … :slight_smile:

– edit: corrected pictures

Interesting. I know lots of folks have built these before, but they are a great way to learn. I am working something very similar to support my vacation home which is 2000 miles away. My aim is to measure and extend the information up to the web and then make it available through my mobile device. The architecture is here (tried to make the picture larger but had to attach it instead):

I have a fair amount of the Arduino code written to measure environmentals, mechanicals, occupancy, events and alarms, etc. I was using the Uno, but stepped back and rewrote it for the Due. I am back up and running with most of that original functionality, and just tonight finding some time to continue work.

The Pi is intended as the consolidation layer within the house. I had spent a fair amount of time trying to get my Arduino’s reliably talking directly to a web server. I had them calling PHP scripts, which loaded attached data to MySQL on the webserver. But it seemed a bit too fragile, so I decided to add the Pi locally as the consolidation point which would then can use a more robust transfer mechanism up to the webserver. Tonight I implemented a protocol between the Arduino and Pi so that I can manage a rather diverse set of commands and parameters between the Arduinos and the Pi. Next on my list is fixing the data architecture and replacing data ‘stubs’ with the real deal.

As far as the web layer goes, I am expecting to have the primary MySQL database there. (Local DB will continue to remain, primarily for consolidation multiple Arduinos, and maybe a local display for troubleshooting.) I have tried a couple JavaScript frameworks, and believe I have settled on Sencha EXT JS. Tons of functionality. And I am tempted by their visual designer and may spring the money for that when the times comes.

Very good fun. A rather slow process due to priorities, but satisfying regardless. I head out to the house for a vacation in February, and am hoping to have something proven enough to leave behind at that time. The clock is ticking!

Best of luck,

Jay

We're thinking along the same lines :slight_smile:
I had drawn something like that before, a bit old-ish, but ...

I might drop the sql/webserver on NAS idea though, it seems more logical to have everything contained on the RasPi. On the other hand I have some experience with the raspi eating two of my SD cards, and the loss of data wouldn;t be all that nice...

Makes sense regarding the SD card on the Pi. I don't have much experience running MySQL on it and what that might do to the life of the card. So the NAS box makes sense. Can you have MySQL installed on the Pi and simply point MySQL to the actual database stored on the NAS box? I.e. split the application logic from the data storage? Given how far away I am from the Pi, I will only be able to physically fix it a few times a year so that might be a good approach.

My next steps is to lock down all aspects of data. Have you decided yet how you will manage the various data types from multiple sources? I.e. one large table, tables for each type of data, tables for each sensor, etc? One thing I am pushing for is to be rather crisp about timestamps. I have divided the day into 14,400 'tenths' of a minute and every record will be time-stamped with a slot. My intention is to store a small amount of data on the Arduino until retrieved and confirmed by the Pi. Validation is basically the Pi assuring it has a reading for each slot. If not it will request it again before deleting from the Arduino. Might be a bit pedantic to worry about each record instead of the bigger picture but I am trying to avoid big chunks of missing data, manage network glitches and power outages, etc...

I was wondering how far along you are on the PHP code? I had previously written some rather simple PHP scripts that could be called via HTTP with parameters attached in order to load the data into MySQL. Is there a pre-built library that I should be considering? I am actually exploring a slightly different approach to move data from the Pi to the web server. I am considering dumping the new data from MySQL to a file on the Pi and then using FTP in batches to move it to the web server. So the PHP code on the webserver would simply detect the new file, and load it into MySQL. Not sure the easiest/best approach to get something up and running quickly.

Thanks for any further thoughts,

Jay

grantnlee:
Can you have MySQL installed on the Pi and simply point MySQL to the actual database stored on the NAS box? I.e. split the application logic from the data storage?

Just create a share on the nas/san/server where you want to store your data, and tell MySQL to store the data there. Mount it in the Rpi (use NFS).

grantnlee:
My next steps is to lock down all aspects of data. Have you decided yet how you will manage the various data types from multiple sources? I.e. one large table, tables for each type of data, tables for each sensor, etc? One thing I am pushing for is to be rather crisp about timestamps.

To be honest: not yet. Not for a single split second. :slight_smile:

grantnlee:
I was wondering how far along you are on the PHP code?

Just a simple script that pipes the data to the db, nothing more. I'm more into hw than software. When I get it up and running with acceptable range, and when I het the low power use I want (ok, ok, that is software based) I'll get busy with the data models and PHP code. If possible get it integrated into a already existing home automation system, but that's for later :slight_smile:

[/quote]

Thanks for the advice. I will give that MySQL storage idea a try.

Knowing you are focusing much attention to the power consumption of your Arduinos, you might check out the Archives at JeeLabs http://jeelabs.org/2013/as they seem to be the best reference for how to drastically cut down on Arduino sensor network power consumption. Tons of time invested and a very thoughtful and well written blog. Unfortunately, he just hung up his hat this fall in order to try out some new things. But still an excellent reference if you have not already seen it.

know that one :slight_smile: