Advice needed on uploading weather data to internet

Other Hardware Home Automation
1

I'm in the process of building a weather station consisting of various indoor and outdoor weather sensors that are connected to an Arduino, to read basic things like temperature (DS18B20), humidity (DHT22) and pressure (BMP180). I might add other sensors later for rain, wind, etc. All sensors are currently connected by wire, and are within several feet of the Arduino.

Eventually I might add an additional indoor sensor station in another room, that will be remotely connected to the Arduino, probably by an nRF24L01+ board--I bought two online but have yet to play with them. But it probably won't get much fancier than that.

Right now I view the most important sensor readings on a 16x2 LCD connected via an I2C backpack, to save on Arduino pins and make things simpler in terms of wiring. I also sometimes view them via a Serial window, which allows me to view all of these sensors' readings, which isn't possible on a 16x2 LCD without paging screens, and I don't want to do that.

I also have an SD shield in the mix, to record all the sensor readings at regular intervals and transfer them to my PC eventually, to be saved either in Excel or a DB.

So far so good, and after much tinkering and teaching myself Arduino, electronics and the specifics of how these sensors work and communicate their readings, and re-leaning how to program in C/C++, I'm fairly satisfied with what I've built.

However, at this point I'd like to add the ability to upload the readings to the internet, to automate getting the data off the Arduino and be able to access it remotely, and to eventually share these readings with others. I first thought about developing a way to simply transfer the data to my PC, via Serial, and then upload the data from it to the internet, but the problem there is that my PC is often in Standby mode, and I want the data to be available continuously, in real time.

So I really do need to be able to connect the Arduino directly to the internet and bypass my PC.

Realizing that, I went and ordered a ESP8266-based breakout board the other day, and should be getting it within a few weeks (yeah, China by way of eBay). In the meantime, I'm trying to learn how to use it. I think I've got a basic handle on how to connect and interface it with the Arduino, in terms of pin connections, level conversion, voltage and current draw, libraries, etc. I'm very new to internet programming so I probably need to learn about TCP sockets, IP protocols, and so on. But I have a programming background so I think I can figure that out.

What I'm really not sure about is "where" I actually upload my data, and how. I'm not looking to turn the Arduino into a rudimentary web server that I can access via a browser. What I want to do is continuously upload the data somewhere "online" where it will be saved indefinitely and securely and be accessible by myself and possibly others whenever and wherever, fairly easily via standard internet programming, that is NOT a web server running on my PC, which it's quite capable of running I'm sure, precisely because it will often be on Standby.

I'm sure that there's a VAST variety of options there, but what are some basic, simple, reliable and most of all FREE ones? I've read about a site called Xively, but it looks too commercial for my needs. I basically just need an online or "cloud"-based place to store a very modest amount of data, that I can both upload data to, and read data from. For all I know Dropbox or Google drive will do the job, but I don't know enough about either to know.

What about WeatherUnderground? I understand that if approved you can upload your weather data to it, and your weather station is accessible worldwide. But can I then programmatically access that data to download it to my PC? Also, I'm looking for a more generic, context-free place to upload my data to, in case I someday want to upload, store and access non-weather data, such as GPS readings, workout info, etc.

I guess I'm basically asking how the "cloud" works on a programmatic and not just end-user basis, from an Arduino/WiFi perspective. I.e. an online "site" where you can store and access data remotely for whatever use, be it to serve data to a web site or phone app, or whatever.

Also, in what "logical" format should my data be stored, to make it fairly universal? Is XML the way to go, perhaps with a weather-specific variation?

2

Check thingspeak as it seems to currently be popular, but there are other server sites too.

3

Thanks. Looks like a definite possibility but at this point I think I'm more interested in learning more about the overall "theory" of how one might go about doing this sort of thing than any specific available solution I might go with. I should have included that in my (already too long) comment.

I.e. what does it mean to want to upload data to the internet from a given device so as to then be able to access it from other devices, and what are the various ways of doing that for a given range of applications?

4

+1 for thingspeak, seems to be the most generous of all the free services.

If you want to understand what goes on 'under the hood' then I suggest using a command-line program like wget or curl to exchange data with some of the sites- that way you can see the structure of the request and the response as well as the different errors that can occur.

If you dont have linux then try sites like http://onlinecurl.com/ or https://www.hurl.it/

Another site you may find useful is http://requestb.in/ which allows you to see the results of an internet request- can be useful to see what your arduino is sending

5

I am in very much the same boat as habanero, minus any programming history, and my project is also very similar and about stable. I know little of the subject, but it is a subject I've been researching for a few months now (multiple hours of every day) and am making little headway. I jumped the gun about a month ago and bought a .com from HostGator because of it's point and click webpage building capabilities of WordPress. So now I am on the hunt to justify that purchase (like $9 per month) of my PHP site by somehow getting my data sent there. WordPress works similar to MS Word which I am much more comfortable on, but that ability is added through plugins.

My most recent studies are suggesting that JSON is the format that the Arduino can utilize. It is unfortunate that I don't have much more. I am also currently trying to find out if "RESTful HTTP" will further streamline the process, but so far I'm of the belief that there are plugins that will enable me to link my .com server to my Arduino project client.

I think we might be looking for the same thing. A .com is a hosted site that is not held to the confines of available micro SD storage space, that my project can always should always forward data to, and also receive commands from. The .com would also allow me to log in from any device on any network and make that device the project's dedicated display.

To the best of my knowledge, Temboo and ThingSpeak are both still too buggy. I don't mean to talk smack, just an observation. Save yourself a few days and skip those site's descriptive pages about their project and go straight into their forums and start reading posts. It seems most people are having some success loading the examples, but the moment people try and graft in new code for new devices, the web pages or whatever start locking up and throwing errors.

6

A few days ago I read an article quoting Steve Wozniak as saying he thinks companies may have overestimated the appeal of connecting everyday objects to the Internet. I've spent a career in information technology and I am always amused at how cyclical information technology trends are. In the 70's and early 80's data was largely centralized, then along came the PC and large amounts of data moved to the desktop. Now the "cloud" offers a centralized "open data platform for the Internet of Things." And so the centralized versus distributed versus localized wheel continues to spin.

Generally, I would recommend that you store data and run apps where they are used the most (apply 80% - 20% rule here.) My home automation / IoT approach has been to have my network nodes (primarily Arduinos moving towards ESP8266s) insert local sensor readings into a MySQL database running on a PC using a WAMPP (Windows, Apache, MySQL, PHP/Python/Perl) stack. When the Raspberry Pi became available it replaced the PC as my network server and runs primarily PHP reporting apps (web pages that can display MySQL data.) I run a cron job on the Pi that hourly gets current and forecast weather data for my locale and inserts it into the MySQL database. I have experimented with using MQTT along with a number of home automation / IoT frameworks and a variety of cloud services such as Temboo and ThingSpeak. My conclusions generally are that all of theses approaches were either too difficult to implement, too rigid, or offered a feature set I don't need.

I thought since the discussion was getting "strategic" I would offer my 2 cents (remember your mileage will very) and so I recommend you continue on your journey to discover an approach / architecture that meets your own particular needs.

Cheers.

7

lafontas:
A few days ago I read an article quoting Steve Wozniak as saying he thinks companies may have overestimated the appeal of connecting everyday objects to the Internet.

I'm surprised that there isn't already a web hosting service to accommodate this market. RPi, Arduino, Beaglebone, Udoo etc etc etc. It'd be awesome if us newbs could buy boards and sensors, plug and play some examples and use a cellphone on a web page to view the results and make commands.

8

myggle:
I'm surprised that there isn't already a web hosting service to accommodate this market. RPi, Arduino, Beaglebone, Udoo etc etc etc. It'd be awesome if us newbs could buy boards and sensors, plug and play some examples and use a cellphone on a web page to view the results and make commands.

I'm guessing that there are such services already, or close to it, with whatever degree of reliability, ease of use and robustness, and it's relative newbies to THIS particular aspect of the broader Arduino/electronics/computer world (like us) who haven't yet found out about it.

There's certainly still a "wild west" aspect to all this stuff that isn't going away any time soon, in that you really do have to plunge in a learn as you go and there aren't enough thorough "how to" guides to help you along--and when there finally are, everyone who's anyone has already moved onto the "next big thing". :slight_smile:

9

lafontas:
A few days ago I read an article quoting Steve Wozniak as saying he thinks companies may have overestimated the appeal of connecting everyday objects to the Internet. I've spent a career in information technology and I am always amused at how cyclical information technology trends are. In the 70's and early 80's data was largely centralized, then along came the PC and large amounts of data moved to the desktop. Now the "cloud" offers a centralized "open data platform for the Internet of Things." And so the centralized versus distributed versus localized wheel continues to spin.

Generally, I would recommend that you store data and run apps where they are used the most (apply 80% - 20% rule here.) My home automation / IoT approach has been to have my network nodes (primarily Arduinos moving towards ESP8266s) insert local sensor readings into a MySQL database running on a PC using a WAMPP (Windows, Apache, MySQL, PHP/Python/Perl) stack. When the Raspberry Pi became available it replaced the PC as my network server and runs primarily PHP reporting apps (web pages that can display MySQL data.) I run a cron job on the Pi that hourly gets current and forecast weather data for my locale and inserts it into the MySQL database. I have experimented with using MQTT along with a number of home automation / IoT frameworks and a variety of cloud services such as Temboo and ThingSpeak. My conclusions generally are that all of theses approaches were either too difficult to implement, too rigid, or offered a feature set I don't need.

I thought since the discussion was getting "strategic" I would offer my 2 cents (remember your mileage will very) and so I recommend you continue on your journey to discover an approach / architecture that meets your own particular needs.

Cheers.

You just gave me an idea, to set up a Pi as a local web server (or some such thing), to avoid having to either use and rely on an external one that either costs more than I want to spend or is too complicated or unreliable or otherwise unappealing or unsuitable for my needs, or keep my PC on 24/7, which I used to do but now find unnecessarily noisy and wasteful of energy (especially in summer where it just creates extra heat). The main downside, aside from the relatively low cost, is that I now have to learn, set up and maintain yet another piece of hardware and software.

I was also wondering whether it's possible for an Arduino to "upload" data to a thumb or hard drive that's plugged into my router's USB port, to which I might be able to grant remote access to in some manner so that I can "serve" this data outside my local network.

More broadly speaking, though, I think it pays to assess one's situation and needs and the various options available before prematurely committing to any given solution path that might end up being a mistake in one way or another, in terms of time, effort, money, aggravation and such. That's why I'm trying to make this discussion more "strategic", as you put it. I'm guessing that in the end I'm probably going to go with a free web hosting service of one sort or another, because I don't really want to get into Pi at this juncture (but would like to eventually), and it'll probably be easier for my fairly simple (or so they seem) current needs.

But first, I want to learn a bit more about how the internet works beyond what I know about it mostly on the client/browser/HTTP side, i.e. on the server side, in its many guises and forms. I.e. what does it mean to "upload" data to a server? What form should the data take as it's sent and then stored, what protocol do you use to do this, and to then access it at a later time? And so on. None of this is unknown or unknowable, but it does require a knowledge of how the internet works that I don't have at present.

I'm guessing that it's not THAT complicated or difficult to learn, at least for my fairly limited needs (no doubt it gets incredibly complex once you start doing fancy stuff like managing large and complex data and serving a large and complex set of clients under commercial conditions). I.e. stuff like TCP, IP, Python, PHP, XML, etc. When I know more about all this, I'll try and post it here FWIW. And thanks for the ideas and advice!

10

myggle:
I am in very much the same boat as habanero, minus any programming history, and my project is also very similar and about stable. I know little of the subject, but it is a subject I've been researching for a few months now (multiple hours of every day) and am making little headway. I jumped the gun about a month ago and bought a .com from HostGator because of it's point and click webpage building capabilities of WordPress. So now I am on the hunt to justify that purchase (like $9 per month) of my PHP site by somehow getting my data sent there. WordPress works similar to MS Word which I am much more comfortable on, but that ability is added through plugins.

My most recent studies are suggesting that JSON is the format that the Arduino can utilize. It is unfortunate that I don't have much more. I am also currently trying to find out if "RESTful HTTP" will further streamline the process, but so far I'm of the belief that there are plugins that will enable me to link my .com server to my Arduino project client.

I think we might be looking for the same thing. A .com is a hosted site that is not held to the confines of available micro SD storage space, that my project can always should always forward data to, and also receive commands from. The .com would also allow me to log in from any device on any network and make that device the project's dedicated display.

To the best of my knowledge, Temboo and ThingSpeak are both still too buggy. I don't mean to talk smack, just an observation. Save yourself a few days and skip those site's descriptive pages about their project and go straight into their forums and start reading posts. It seems most people are having some success loading the examples, but the moment people try and graft in new code for new devices, the web pages or whatever start locking up and throwing errors.

Yeah, most discussions of the Arduino's web/internet capabilities seem to be more about using it as a very basic and crude server, for which it's ill-suited, than as a "client" that uploads data to a "real" server, to then be accessed by things like browsers and apps and such. But HOW does one do this?!? That's what I'm (and it appears you) are trying to figure out. For all I know I might end up going with one of these prefab hosted services, but as per the Arduino ethos I'm kind of a DIY person and wouldn't mind getting into more of the nuts and bolts of how to do this and, well, do it myself. :slight_smile:

11

You may want to start off by installing an Apache/MySQL/PHP suite on your PC since that's an environment your familiar with. XAMPP (https://www.apachefriends.org) and AMPPS (http://www.ampps.com/) are both great.

Next, you`will want to create a database and a table to store your sensor readings. Here is a working example I use:

--
-- Table structure for table `nodenames`
--

CREATE TABLE IF NOT EXISTS `nodenames` (
  `Node` varchar(3) NOT NULL,
  `NodeName` varchar(25) NOT NULL,
  `timeStamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `recNum` int(11) NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`Node`),
  UNIQUE KEY `recNum` (`recNum`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=4 ;

--
-- Dumping data for table `nodenames`
--

INSERT INTO `nodenames` (`Node`, `NodeName`, `timeStamp`, `recNum`) VALUES
('1', 'Office', '2014-10-16 02:57:44', 1),
('2', 'Bedroom', '2014-10-16 02:57:29', 2),
('3', 'Test', '2014-10-16 02:55:35', 3);

-- --------------------------------------------------------
--
-- Table structure for table `sensorLog`
--

CREATE TABLE IF NOT EXISTS `sensorLog` (
  `timeStamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `node` int(3) NOT NULL,
  `temperature` int(11) NOT NULL,
  `humidity` int(11) NOT NULL,
  `light` int(4) NOT NULL,
  `pressure` float NOT NULL,
  PRIMARY KEY (`timeStamp`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

--
-- Dumping data for table `sensorLog`
--

INSERT INTO `sensorLog` (`timeStamp`, `node`, `temperature`, `humidity`, `light`, `pressure`) VALUES
('2015-05-27 00:24:51', 3, 27, 52, 0, 0),
('2015-05-27 00:54:51', 3, 27, 51, 0, 0),

These SQL scripts are used to create 2 tables in the MySQL database. The first table is a lookup table that contains the locations or node names of your client sensors. The second table stores the values uploaded (there's that term) from an Arduino over an ethernet network.

Here is an example of a working Arduino script that takes sensor reading and the uploads them to the MySQL database:

/*
Release Notes:
==============
-
-

To Do List:
===========
- 
-
*/
/*-----( Libraries )-----*/
#include <Time.h>
#include <Ethernet.h>
#include <TimeAlarms.h>
#include <SPI.h>
#include <DHT.h>
#include <Wire.h>
/*-----( Declare Objects, Variables, Constants, Pin Numbers )-----*/
//-----------------------------------
//----- Identification --------------
const byte nodeNum = 3;
const float verNum = 2.09;
//-----------------------------------
//----- Room Temp LED - RGB LED -----
const byte tooChilly = 18;
const byte tooHot = 21;
const byte ledRedPin = 7; // output pin for the LED
const byte ledGreenPin = 6; // output pin for the LED
const byte ledBluePin = 5; // output pin for the LED
const byte fadeAmount = 5;    // how many points to fade the LED by
byte brightness = 175;    // how bright the LED is
//-----------------------------------
//----- Ethernet - TCP --------------
byte mac[] = { 0xEE, 0xBD, 0xBE, 0xEF, 0x03, 0xED };
byte ip[] = { 192,168,0,214 };
byte gateway[] = { 192,168,0,1 };
byte mydns[] = { 192,168,0,1 };
IPAddress server(192,168,0,16); // IP address off the PC/Server/Pi etc. where your MySQL database is located
int serverPort = 80;
EthernetClient client;
String data;
//-----------------------------------
//----- DHT11 - Temp & Humidity -----
#define DHTPIN 8 // SENSOR PIN
//#define DHTTYPE DHT11   // DHT 11 
#define DHTTYPE DHT22   // DHT 22  (AM2302)
DHT dht(DHTPIN, DHTTYPE);
int t = 0; // TEMPERATURE VAR
int h = 0; // HUMIDITY VAR
float avgTemp = 0;
//-----------------------------------

void setup() {
  Serial.begin(9600);
  //-----------------------------------
  Serial.print("Starting testNode ...");  // so we know what sketch is running
  SplashDisplay();
  //-----------------------------------
  Serial.println(F("Starting ethernet TCP ..."));
  Ethernet.begin(mac, ip, mydns, gateway);
  Serial.print("Static IP number assigned is ");
  Serial.println(Ethernet.localIP());
  //-----------------------------------
  Serial.println(F("Starting DHT11 Sensor..."));
  dht.begin(); 
  delay(5000); // GIVE THE SENSOR SOME TIME TO START
  //-----------------------------------
  Serial.println(F("Temp LEDs..."));
  pinMode(ledRedPin, OUTPUT);
  pinMode(ledGreenPin, OUTPUT);
  pinMode(ledBluePin, OUTPUT);
  //-----------------------------------
  Alarm.timerRepeat(60, MinuteRepeats);     // timer for every 60 seconds    
  Alarm.timerRepeat(1800, HourlyDataUpload);// timer for every 360 seconds    
  //-----------------------------------
  MinuteRepeats();
  HourlyDataUpload();
}

void loop() {
    Alarm.delay(1000); // wait one second between clock display
}

/*-----( Declare User-written Functions )-----*/

void MinuteRepeats(){
  Serial.println("--> MinuteRepeats Function");
  ReadSensors();
  RoomTempLED();  
}

void ReadSensors() {
    Serial.println("--> ReadSensors Function");
    t = (int) dht.readTemperature();
    Serial.print("Temperature: ");
    Serial.print(t); //display 2 decimal places
    Serial.println(" deg C");
    h = (int) dht.readHumidity();
    Serial.print("Humidity: ");
    Serial.print(h); //display 2 decimal places
    Serial.println(" %");
    //-----------------------------------
    avgTemp = t;
    Serial.print("Average Temperature: ");
    Serial.println(avgTemp, 2); //display 2 decimal places
  
    Serial.println();//line break
}

void RoomTempLED() {
  Serial.println("--> RoomTempLED Function");
  Serial.println("");//line break
  Serial.print("Average Temperature: ");
  Serial.println(avgTemp, 2); //display 2 decimal places
  if (avgTemp <= tooChilly) {
    analogWrite(ledBluePin, brightness);
    analogWrite(ledGreenPin, 0);
    analogWrite(ledRedPin, 0);
  } else if (avgTemp > tooChilly && avgTemp <= tooHot){
    analogWrite(ledBluePin, 0);
    analogWrite(ledGreenPin, brightness);
    analogWrite(ledRedPin, 0);
  } else if (avgTemp > tooHot) {
    analogWrite(ledBluePin, 0);
    analogWrite(ledGreenPin, 0);
    analogWrite(ledRedPin, brightness);
  }
  //-----------------------------------
  Serial.println();//line break
}

void SplashDisplay(){
   Serial.println("Home Automation");
        Serial.print("Node: ");
 Serial.print(nodeNum);
        Serial.print(" - Ver ");
 Serial.println(verNum);
}

void HourlyDataUpload() {
    Serial.print("--> HourlyDataUpload Function ");
    data="";
    data+="temp1=";
    data+=(int) avgTemp;
    data+="&hum1=";
    data+=(int) h;
    data+="&node=";
    data+=(int) nodeNum;

    if (client.connect("192.168.0.16",80)) { // REPLACE WITH YOUR SERVER ADDRESS
      client.println("POST /WebClient/add.php HTTP/1.1"); 
      client.println("Host: 192.168.0.16"); // SERVER ADDRESS HERE TOO
      client.println("Content-Type: application/x-www-form-urlencoded"); 
      client.print("Content-Length: "); 
      client.println(data.length()); 
      client.println(); 
      client.print(data); 
    } 

    if (client.connected()) { 
      client.stop(); // DISCONNECT FROM THE SERVER
    }
    
    Serial.println("==========================");
    Serial.print(" - ");
    Serial.println(data);
    Serial.println("========================================================");
}

/* ( End ) */

This Arduino script takes reading from a DHT11/22 temperature/humidity sensor and passes them to a PHP script called add.php which takes care of actually inserting the passed values into the MySQL table.

Here is a working copy of add.php:

<?php
   include("connect.php");
   
   $link=Connection();
 
 $node=$_POST["node"];
 $temp1=$_POST["temp1"];
 $hum1=$_POST["hum1"];
 $pcel1=$_POST["pcel1"];
 $press=$_POST["press"];

 $query = "INSERT INTO `sensorLog` (`node`,`temperature`, `humidity`, `light`, `pressure`) 
 VALUES ('".$node."','".$temp1."','".$hum1."','".$pcel1."','".$press."')"; 
   
   mysql_query($query,$link);
 mysql_close($link);

   header("Location: index.php");
?>
-------------------------------
connect.php

<?php

 function Connection(){
 $server="localhost";
 $user="mysqlusername";
 $pass="mysqlpassword";
 $db="mysqldatabasename";
   
 $connection = mysql_connect($server, $user, $pass);

 if (!$connection) {
     die('MySQL ERROR: ' . mysql_error());
 }
 
 mysql_select_db($db) or die( 'MySQL ERROR: '. mysql_error() );

 return $connection;
 }
?>

In terms of hardware this example uses a PC/server/Pi running the Apache web server, MySQL and PHP. It is connected to an Arduino with a W5100 ethernet shield and a DHT11/22 sensor. Hope that helps move on down the road. Cheers.

12

Thanks, this helps, although I'm looking to avoid hosting the data server on my PC because it's often on Standby/Sleep and I want whatever I upload sensor data to to be up and running 24/7.

I can certainly store readings from my Arduino-based weather station if the server is unavailable, on an SD card or EEPROM chip, and perhaps even on a thumb drive plugged into my router's USB port, to be uploaded when the server's back online. But I also want to be able to serve that data at all times, initially to myself so I can access it, be it from my PC, laptop, smartphone or such, and eventually to others. Using my PC as a server makes that impossible at present.

However, the basic solution you presented is I'm sure applicable to online server services I may end up using. No doubt many if not most online server services use some variation of this approach. Hopefully some are free!

I do like this method over the one that some sites use, like ThingSpeak, where they add an additional layer of protocol and code over it to make it more "user-friendly". This might make more sense for people who don't have much programming and DB experience and just want a place to dump and retrieve their data, but it adds additional back-end overhead and the potential for problems. Plus, it's not as fun. :slight_smile:

I'd rather cut out the overhead and talk directly to the DB, that I can also structure any way I like.

Do you know if there are sites that allow you to set up and run your own DB that can receive and serve data to the outside world, that maybe offer free options so long as your DB size, data sent back and forth and processing capacity used don't exceed a certain amount?

I.e. free data and application servers?

Thanks again.