Standalone Datacollecting Webserver

I collect and store data on cosm for my house; have been doing this for a few years now. I have an arduino 2560 that acts as a collector for the various sensors and controls around the house and I send some of the data up to cosm, emoncms, and thingspeak. The reason I send it to all three of these is that I wanted to try out as many of the cloud storage services for this kind of thing as I could. The arduino collector also serves as a web server for control and monitoring of the house. I can load its web page and it will tell me what's going on and I can change things if I want to.

I put together some json code and it runs in the user's browser to present a graph of temperature and power usage for a 24 hour period. I did it this way because I wanted it to be as close to real time as possible and I wanted to be able to expand the graph enough to see what appliances were running (you get to know them intimately over time). The cool thing about using the user's browser is that the arduino doesn't have to do any work, it just sends code to the browser. The json code is held on a cloud server as well so that it doesn't take up space on the arduino.

You can see the screen I put up to monitor the house, but remember, it's an arduino with 4 simultaneous connections, so you may have to try a couple of times to actually get in. Once you get in and the page starts to load, the data for the last 24 hours is gathered and presented on your browser, so it may take 10 seconds or so to collect, and sometimes cosm will time out first. The timeout doesn't happen very often so it doesn't bother me. If you hit it with a cell phone, the little browsers on them sometimes don't like the iframe that holds the graphs and gauges, just use a different browser or use a pc.

This is very much a work in progress and probably will be for a long time. I change it pretty often, mostly because I can, but sometimes I add or change something based on experience. The little arduino is at <link>, but be kind to it, it's only a little arduino, not a bank of servers like google or bing.

Oh, I also write about this setup on my blog, see the signature line for the link.

So yea got my temperature being graphed on my website so now I have to get the other two working so does anyone know if you can send data of 3sensors to cosm? or is there away to graph all three onto a single graph?

You can send the data from a lot of sensors to cosm. I send several to them. The provide an easy way to get graphs, one data set at a time. If you want to graph temperature and humidity, it's easy to send them the data and get graphs for each of them. If you want both of them on the same graph, it gets harder. The graph I have is temperature and power usage; that took some javascript code to do. I just used the google graph api and some javascript code to do mine.

Take an actual look at cosm, they have a ton of articles on how to send the data and get it back to do something with.

yea got my temperature being graphed on my website! so more less on my way, ok cool thanks :stuck_out_tongue: , so u can use 3sensors sent to cosm so does that mean in the arduino IDE i have to set up extra 2 feeds and API yea?

You need to distinguish between feeds and datastreams.

Your API key is your personal cosm-usage pin which I understand can be used for several cosm feeds, amongst other things.

The feed id is for a specific feed display. It is to identify that display and the display can have several datastreams.

The datastream is the individual sensor, calculation, or information. If you just want to add a new datastream, you simply change the line in your programme

CosmFeed feed(83153, datastreams, 4) - in this case four streams and configure the extra stream. Cosm does the rest.

Im using the wifi shield at the moment, using the arduino wifi pachube example added an extra sensor

Add two reading lines at the start of loop(), ie

int sensorReading = analogRead(A0);
int sensorReading2 = analogRead(A1);

Pass the 2nd reading into the 'sendData' method at the end of loop()

sendData(sensorReading, sensorReading2);

Then change the sendData method to accept two values and package those up for sending also

// this method makes a HTTP connection to the server:
void sendData(int thisData, int thisData2) {
// if there's a successful connection:
if (client.connect(server, 80)) {
Serial.println("connecting...");
// send the HTTP PUT request:
client.print("PUT /v2/feeds/");
client.print(FEEDID);
client.println(".csv HTTP/1.1");
client.println("Host: api.cosm.com");
client.print("X-ApiKey: ");
client.println(APIKEY);
client.print("User-Agent: ");
client.println(USERAGENT);
client.print("Content-Length: ");

// calculate the length of the sensor reading in bytes:
// 8 bytes for "sensor1," + number of digits of the data:
// plus the same again and length of data 2
int thisLength = 8 + getLength(thisData) + 8 + getLength(thisData2);
client.println(thisLength);

// last pieces of the HTTP PUT request:
client.println("Content-Type: text/csv");
client.println("Connection: close");
client.println();

// here's the actual content of the PUT request:
client.print("sensor1,");
client.println(thisData);
// and the 2nd sensor
client.print("sensor2,");
client.println(thisData2);

}
else {
// if you couldn't make a connection:
Serial.println("connection failed");
Serial.println();
Serial.println("disconnecting.");
client.stop();
}
// note the time that the connection was made or attempted:
lastConnectionTime = millis();
}

this code i found on the cosm forum

this is the message I get
and i get no values on my feed

CSV Parser Error: CSV is invalid. Incorrect number of fields
disconnecting.
connecting...
HTTP/1.1 400 Bad Request
Date: Sat, 09 Mar 2013 14:00:22 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 60
Connection: close
X-Request-Id: 94849a7b1e96abaa62c7145ff139221428ceddcb

I hesitate to say this because you are using WiFi and there may be something going on I don't understand. But.

I guess cosm doesn't know or care how you send the data, so long as the commands are right.

And

I see the word useragent

And you don't get a connection. So it would appear you need to use the latest com libraries. The code is incomplete, so the libraries you are using is unclear.

ahh ok cool thanks will look into that, I also have the Ethernet shield if you have any more advice or solutions/ code for that it would be great!

OK, now that you haven't summarily dismissed my comment, my guess is that it is definitely your problem and explains why your code looks like incomprehensible junk.

I get the impression that cosm underwent a palace revolution about last September and, unless you are one of the gold star brigade, anything you pick up on this matter that is older than that, is just going to cause you grief.

Check the cosm site, get the kosher libraries, cosm.h and httpclient, and you will probably be fine. You will see examples in the "It Worked!" section of the cosm forum, including one of mine, and there is stuff included with the libraries.

I do hope your ethernet shield is WIZ5100 based........

Hi again thanks I got the wifi shield working and reads and graphs the sensors thank you for that needed the libraries

I do hope your ethernet shield is WIZ5100 based........

Yes it is!
Now I am trying out the ethernet shield but this doesnt connect its fairly similar code to the wifi

#include <SPI.h>
#include <Ethernet.h>
#include <HttpClient.h>
#include <Cosm.h>

// MAC address for your Ethernet shield
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

// Your Cosm key to let you upload data
char cosmKey[] = "YOUR_COSM_API_KEY";

// Analog pin which we're monitoring (0 and 1 are used by the Ethernet shield)
int sensorPin = 2;

// Define the strings for our datastream IDs
char sensorId[] = "sensor_reading";
CosmDatastream datastreams[] = {
  CosmDatastream(sensorId, strlen(sensorId), DATASTREAM_FLOAT),
};
// Finally, wrap the datastreams into a feed
CosmFeed feed(15552, datastreams, 1 /* number of datastreams */);

EthernetClient client;
CosmClient cosmclient(client);

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  
  Serial.println("Starting single datastream upload to Cosm...");
  Serial.println();

  while (Ethernet.begin(mac) != 1)
  {
    Serial.println("Error getting IP address via DHCP, trying again...");
    delay(15000);
  }
}

void loop() {
  int sensorValue = analogRead(sensorPin);
  datastreams[0].setFloat(sensorValue);

  Serial.print("Read sensor value ");
  Serial.println(datastreams[0].getFloat());

  Serial.println("Uploading it to Cosm");
  int ret = cosmclient.put(feed, cosmKey);
  Serial.print("cosmclient.put returned ");
  Serial.println(ret);

  Serial.println();
  delay(15000);
}

obviously i add all the information that is needed to be added
so
In the serial display it now showing Error getting IP address via DHCP, trying again...
any solution to this would be great

Try setting an explicit IP address instead of relying on DHCP.

Will give that a try thanks for the pointers

I'm not sure what you have and have not done. Are you saying that you have everything working OK, with the single sensor graph displaying at cosm, when you connect to your router by WiFi, but when you try to do the same thing by ethernet to the same router, it will not work?

Your code is more or less the same as mine, and should work. The only discrepancy I can see is the buffer command missing

char sensorId1[] = "OutThermo";

const int bufferSize = 100;
char bufferValue[bufferSize]; // enough space to store the string we're going to send
CosmDatastream datastreams[] = {

The failure appears to be in the setup section but it is essentially the same as mine.

Yes that is basically it! I have the 3 sensors working fine with the wifi but has no buffer :confused: I haven't done much to change the code from wifi to Ethernet. Will try work on it tomorrow and let you know what happens, thanks!

Well, that's a relief. I'm assuming the WiFi is the hard bit. I can't see what the problem is but it is surely something quite trivial.

Hmmmm now its sayinig that its connected and uploading to cosm but for some reason its not doing it, no sensors values are graphed must be something small in the code, it's getting better will inform if it works!

Are you seeing the ret when you put to cosm? ret = 200 means a kosher transfer. If you get that but see nothing in the graph, it could be the data is suss. You can test just before you send

Serial.println(datastreams[0]);

Serial.println("Uploading it to Cosm");
ret = cosmclient.put(feed, cosmKey); // SEND FEED TO COSM
Serial.print("cosmclient.put returned "); // 200?
Serial.println(ret);

Yes have that part of the code alright,

This is what is displaying in the serial monitor

Starting muiltiple datastream upload to Cosm...

Thermister 22.79
relativeHumidity 26.75
BPW34 723.18
Uploading it to Cosm
cosmclient.put returned -3

Looking at the cosm debug info nothing showing too its still Awaiting incoming requests...:confused:
strange that wifi shield works but not the Ethernet shield

ahhhh I got it now working! nothing wrong with it just isn't working in college! works perfectly at home :smiley: happy days,

By any chance do you know if there is a wifi signal booster for the wifi shield just chancing my arm or does anyone, might try tin foil or something hehe

No he-he about it. My neighbour uses foil food tray reflector to boost his WiFi signal. To good effect, I understand.
There are bigger antenna rods available on eBay for just a few dollars, but these are for the router. I don't know how you would beef up a WiFi shield but I bet it can be done. Failing that, you might be able to use a router as an intermediary.