next topic »

[on:]

So I have this big set of data on Cosm/Pachube that I want to automatically take the area under the curve during specific time frames. This calculation should be performed once a day, at the end of the day. Now, I am perfectly aware of how to do this with Excel using the area of trapezoid formula, but it takes manual effort. Automatic would be key for my application. I want to do this every day and store the timestamp it was performed and area under the curve value.

I am trying to brainstorm different ways to do this. The two things I could come up with are:

-GET a single day's worth of data from cosm and store it in a server side database. Then some how do the area under the curve calculation and store it's value in the same database.

-Do an "on the fly" type calculation inside some type of programming language. GET each single data point from Cosm, store it temporarily in a variable while doing a calculation then load in a new variable and keep going until the last datapoint is calculated. Then store that number in an array that has the date created and the value.

I don't know much at all about databases and would like to start with a very basic database that would be easy to learn (if that is needed). I just want something that I can calculate the area under the curve and look at the data on a website everyday.

If there is an easy way to dissect the Cosm data that I don't know about. Or if someone could make some very general/basic suggestions on how to start with this, that is what I'm looking for. Many thanks in advance and please forgive me if I haven't done enough homework before posting here.

[Reply #1 on:]

if you have an Y value every 10 seconds you can just summerize them, in short it should look like this code:

Code:

void setup()
{
  sum = 0;
}

void loop()
{
  if (10 seconds passed)
  {
    Y = measurement();
    sum += Y * 10; // value Y * 10 seconds
    processY(Y);
  }
  if (day has passed)  // or after 20 loops or after an hour whatever
  {
    processSum(sum);
    sum = 0;
  }
}

If Y is speed then sum would be distance.

Trapezoid addition tosum is also possible; you just need to keep the previous value Yprev, same type as Y

Code:

...
  if (10 seconds passed)
  {
    Y = measurement();
    sum += (Y/2 +Yprev/2) * 10; // trapezoid
    Yprev = Y;
    processY(Y);
  }

advantage of on the fly processing is that you constantly have an up to date value / indication of the area. This means if it exceeds some value you can act (near) real time

[Reply #2 on:]

Many thanks Rob. So you are saying this calculation can probably all be done inside the Arduino. No need to start a database or anything. I would just upload the calculated area over to Pachube/Cosm once a day.

Just thinking of some problems that might occur with that. If I lose power to the device then the on-the-fly calculation would be lost? I guess I would have to write the value to EEProm and if the device turned off then load in the old value and continue on. I am guessing the calculation will still be fine given that you lost the x-value (time)?

[Reply #3 on:]

So you are saying this calculation can probably all be done inside the Arduino

this calculation is not a problem ,but I don't know what the arduino should do more ...

No need to start a database or anything.

Not directly, but a DB with your sensordata is often no bad idea as it allows you to change the view on the data.
Maybe you want the area over the last hour or in chunks of 15 minutes ...

I would just upload the calculated area over to Pachube/Cosm once a day.

I would say just as often as you upload the other data - or do you not upload those?

If I lose power to the device then the on-the-fly calculation would be lost?

Yep, unless you add an SD card to the Arduino, to store raw measurements and area math results. Add a RTC to get the thing robust.
Have a look at this shield - http://www.ladyada.net/make/logshield/index.html -

I guess I would have to write the value to EEProm and if the device turned off then load in the old value and continue on.

I would go for SD card, depends how often you write in fact. 
If you should continue depends on the duration of the black out. If it is for 3 days you cannot continue

I am guessing the calculation will still be fine given that you lost the x-value (time)?

see RTC tip above

[Reply #4 on:]

Here is my feed:

https://cosm.com/feeds/30326?pachube_redirect=true

I take a measurement once every 2 minutes or so and then upload the 4 data points you see to pachube. Basically I want to take the area under the curve of "dirty" and "clean".

Suppose I wanted to do this calculation all server-side instead of how we are discussing on the Arduino. I was thinking that I could just PULL data from Pachube, do the calculation, then PUSH data back onto Pachube on a completely new graph. If that was the case I suppose I could just write some script to do that for me. What programming language to do it in? Perhaps PERL or something that handles the data set well.