HOWTO for simple monitoring and WWW graphs

Hi all

I just wrote this quick HOWTO on simple monitoring and graphs using collectd and rrdcgi.

Hope this helps!

nice one!

Very interesting, I'm going to use this on my current project:Distributed communications ideas?

I think it will be just the ticket for the data collection/storage/display portion of the project. It took me a while to have the time to actually try it (work so often gets in the way of fun!) But I finally did it.

I had one stumbling block along the way: when I first brought up the rrd.cgi page, I got error messages about not being able to open the rrd data files. But looking at the indicated path, it seemed like they were there. Finally, I noticed the path in the error message was "/mnt/sda1/data/collectd/rrd/Arduino/memory/memory-free.rrd" whereas my data was at "/mnt/sda1/data/collectd/rrd/Arduino2/memory/memory-free.rrd"

It was easy to miss the extra "2" in the middle of the pathname.

It might be worth it to mention in your HOWTO that the code in the rrd.cgi file assumes that the name of your Yun is the default "Arduino" and it must be updated if you've change the name of your Yun (as I had done.)

If I had used a drastically different name, I probably would've noticed the discrepancy much sooner.

Thanks for your post and HOWTO page! It looks like it's going to be very helpful to me. Now I have to figure out how best to implement a custom input plugin for the data that my board collects...

If I might ask, did you use a tool to create the page in the link or did you code it by hand.
Since I see so many how-to pages in this format, I was wondering if there was a tool to create it.

Please pardon me for being a little off subject.

I've got this example running on my Yun, and I like what I see so far. 8)

My next step is to get some custom data into the RRD database. From my reading, it looks like using collectd's python plugin might be the most efficient way of handling it. (There is also the exec plugin, but that requires spawning a new process for each update, the python plugin does it without needing all of that overhead.)

My problem is that collectd can't find the python plugin: when starting collectd it prints out "Could not find plugin python." then the logfile shows a bunch of errors indicating that it doesn't know what to do with the python plugin configuration items (because it couldn't find the plugin!)

The python plugin is said to be available in collectd starting in version 4.9. When opkg installs collectd it's version 4.10.7, so I guess it should be available?

None of the output of these commands list anything that sounds like it would be the plugin (I would assume it to be named collectd-mod-python?)

opkg update
opkg list | grep collectd
opkg list | grep python

Is the python plugin for collectd available for the Yun? Where can it be found?

Looking at the collectd makefile (github) it seems that the python plugin was disabled by OpenWRT mantainers.

Can you plase open an issue on Github so we can track it for future updates?

Angelo9999:
Looking at the collectd makefile (github) it seems that the python plugin was disabled by OpenWRT mantainers.

Hmmm, there must have been a reason for not including it. Seems some of the other language versions of the generic plugin are also disabled, like perl and java. Maybe something in there is difficult to port?

Can you plase open an issue on Github so we can track it for future updates?

I've never done anything with Github other than to look at an occasional file. Guess I have some learning to do - yet another thing to add to the list.

In the meantime, I've found a very useable workaround: http://forum.arduino.cc/index.php?topic=238693.msg1948386#msg1948386

Sorry all, I didn’t reply as I wasn’t getting post notifications. I will answer some of these questions asap!

I assume you’ve been using it for a while, how would you characterize the reliability? I’ve been having problems where collectd will just stop collecting data. Sometimes rebooting brings it back, sometimes not.

I haven’t gotten too serious with it, just running some long term trials. If it isn’t reliable, I don’t really want to put it in “production” so to speak.

Hi all

In answer to the questions:

  1. Yes, I should update it to highlight the name, sorry it cost you some cycles :frowning:
  2. The format used for the page is "Markdown" .md which is understood natively by github and rendered into an HTML format automatically. It's a very old method of inline formatting for text used before html was around, and re-gaining popularity as a quick and simple formatting "language". To use it on Github just checkin a file in markdown format with a .md extension and github will display it.
  3. OpenWRT presumably disables the plugins because they're not usually wanted on memory constrained embedded systems. I tend to use exec and the shell for custom reporting - I will update the page shortly with an example
  4. Reliability has been fine for me; make sure you're using the latest build of the software (see sticky at top of forum) and also verify that your SD card isn't loose. If you lose the file system then collectd can die. My main issue is that my Reptile House is using a wifi link to the main house and I sometimes lose UDP packets when that link suffers from interference. Since collectd networking uses UDP by default, you will occasionally suffer from loss of packets. My complete architecture is DS18B20 -> arduino -> REST bridge -> yun linux collectd (with local RRD) -> OpenWRT WRTLS54G in NAT client mode with collectd forwarder -> ASUS Chromebox reflashed to Ubuntu -> collectd server -> carbon (whisper) -> graphite. I would also check your sample rate and reduce it if you don't need fast sampling.

(also still not getting email notifications... looking into that)

Oh, and one point to note - the exec plugin only spawns one process. It keeps it open and just reads it continuously - see my updated example. So there is no need to worry about it spawning a new shell for every update.

cydergoth:
4) Reliability has been fine for me; make sure you're using the latest build of the software (see sticky at top of forum) and also verify that your SD card isn't loose.

I think time is a big issue with my data loss. With no RTC, the Yun needs to get the time over the network every time it boots. I suspect that collectd is starting before the time is valid. Given the way collectd works, rejecting all updates with a time stamp that is the same or earlier than the last sample, this can cause problems: if it gets a bad sample with a time in the future, all subsequent samples are rejected until the real time catches up with this bad time. With a true RTC this shouldn't be an issue, but without one I've seen some "interesting" times when the Yun doesn't immediately get a valid date. (I'm still trying to confirm this theory.)

(also still not getting email notifications... looking into that)

I've not received any notifications since the recent forum software upgrade. :frowning:

cydergoth:
Oh, and one point to note - the exec plugin only spawns one process. It keeps it open and just reads it continuously - see my updated example. So there is no need to worry about it spawning a new shell for every update.

I've seen the comment about spawning exec multiple times in several places. In those examples, the code collected one sample, then exited. With code structured like that, collectd will indeed spawn a task for each sample. I have since found that if the spawned script loops and sends multiple samples, it will not be constantly re-spawned. (It will be re-spawned if it exits for any reason, perhaps because it crashed.)