Suitable WEB Technology Choice for Remote Arduino Data Visualisation

Hi All,

I have a range of simple sensors that I have working via an Arduino Uno and a Due. I am making a proof of concept but am struggling to select a suitable web technology for the backend.

Goal:

  • Collect data from sensors (operating 24/7 or on event)
  • Transmit using EthernetShield
  • Receive via backend web technology, ie .PHP or Node.js etc
  • Format, present & visualise using Javascript
  • Have data available for use thereafter

I have done a lot of reading both here and the wider Internet and it seems that there is a huge range of tutorials/code for the Arduino => Node.js using Serial comms, which I do not want to do.

Does anyone have a suitable alternative? I had considered that Node.js may be a great option but it seems that it's complexity or perhaps newness to me, may be an obstacle.

Anything considered and there is no requirement for a heavy duty DB at the web server/service end, but I'll consider it if required.

TIA, QH.

qh881:
Goal:

  • Collect data from sensors (operating 24/7 or on event)
  • Transmit using EthernetShield
  • Receive via backend web technology, ie .PHP or Node.js etc
  • Format, present & visualise using Javascript
  • Have data available for use thereafter

Does anyone have a suitable alternative?

Why not just send the stuff to Internet of Things, like Xively?

That probably fixes Goals 1,2, and 5, but I'm afraid I have never heard of "backend web technology" and have no idea why you would need to "Format, present & visualise using Javascript", so maybe you have different needs.

Thanks for the reply Nick.

By "Backend web technologies" I meant .PHP, .NET, Node.js, any framework etc.

By "Format, present & visualise using Javascript" I meant make pretty pictures, graphs and graphics (dynamic Canvas elements).

... clearly I am not a web nor application developer, so please excuse my misuse of terms or lack of understanding of said field.

Xively, eh? I'll look into it now.

QH.

+1 to Nick's comments but I find Xively to be very commercial these days so my later streams are stored with
https://thingspeak.com/ I've heard good things about https://grovestreams.com/ too but never got beyond registering for a free account.

Thingspeak is nice because you can update your data with a http GET which is often easier than the traditional POST because you dont have to mess around with headers or calculate content size.

There are lots of other such services on the web, just make sure you understand the limits on how much/how often you can store data. Typically they limit you based on some combination of total size of data, length of time data is retained or update rate.

Thanks again for that, yes Xively does seem commercially slanted but thingspeak looks interesting. I have 15 sensors in my home, all polling at least 1 event per second, some more, 24/7 ... so good call re considerations.

+1 for Thingspeak.

  1. Most annoying limitations:
  • Do not expect to upload data (POST) faster that 15 seconds.
  • 8 Data Fields per Channel. But you can do more than one channel if needed.
  1. Nice thing: very simple Channel feed, send an HTTP GET or POST to
    https://api.thingspeak.com/update

POST example:

POST https://api.thingspeak.com/update
api_key=YOURAPIKEY
field1=123

The response will be the entry ID of the update, for example: 14

If the response is 0 then the update failed.

  1. Very, very nice thing: Source available, if you don't like your data over the Internet just install your own Local Thingspeak Server.

If you want to see a step by step install even for a small Raspberry PI running Thingspeak Server, you can take a look here: RPI Thingspeak Installation instructions

Youtube Video:Raspberry PI 2 - Thingspeak Server for ESP8266 Battery Monitor System - YouTube

qh881 wrote:

I have a range of simple sensors that I have working via an Arduino Uno and a Due

Can you describe better what your actual setup is or what you have in mind to do, in terms of what is the function of each of the Arduinos, both UNO and DUE?

Which Arduino will take the sensors ?
Which Arduino will have the Ethernet shield ?

Then in terms of your needs of being able to collect data from sensors, can you clearly describe this ?
What are your needs or wishes in terms of being able to see real time values from sensors, such as update period ?
What is your need or wish in terms of logging sensor values to a historical database, again, how often ?

Is there any control, or is this simply a monitoring project ?

You wrote:

I had considered that Node.js may be a great option but it seems that it's complexity or perhaps newness to me, may be an obstacle.

What is your skill level in terms of understanding and generating code for the various aspects of server and client sections, for example, PHP and javascript ?

trackerj makes a good point with some of the things to understand and watch for if considering an online service to stream your data to for both storage and visualisation. You might want or need your data at different rates of update and you might wish to be free to generate your own visualisation.

In another thread in this section I posted a comment that may be of some value for you to gain some further ideas, some more thoughts

There is a point where a project crosses over from being basic and simple where you can simply use your Arduino as the main HTTP connector, either as server or as a HTTP feeder into a HTTP server, to a point such that your project requires more grunt, features and resources found in using a small GNU/Linux board, such as RPi, or Beagle whatever, or any number of these ARM based boards.


Paul

trackerj:
+1 for Thingspeak.

  1. Most annoying limitations:
  • Do not expect to upload data (POST) faster that 15 seconds.

Yes, I should have noted that you need to check the update frequency. Xively is good for ten second updates, which is the same frequency that I log to SD. Local display is 1 sec. I have not actually used Xively for a while but, last time I looked, my old Cosm stuff still worked. I got pissed off with Xively because their forum went sour but I have stayed with it because of the frequency and think their displays are better than the others I have seen.

Like rw95.... I've heard good things about grovestreams but never got beyond registering for a free account.

Have you considered using Meteor, https://www.meteor.com/
Hope you have success.

Thanks to all for those great ideas, I'll work through them and evaluate as per my needs.

As an update, I have got the Arduino Uno + EthernetShield + DS18B20 OneWire circuit talking to a basic Node.js app ... it's ugly and I need to make it pretty & route correctly but that's progress. Ultimately, a remote, hosted service might be the way to go.

Any thoughts/comments from anyone in relation to the BananaPi & BananaPi Pro that are available ex-China now for same price point as RPi?