Arduino Forum

Development => Other Software Development => Topic started by: robtillaart on Jan 30, 2011, 07:01 pm

Title: RunningAverage Class on playground (updated)
Post by: robtillaart on Jan 30, 2011, 07:01 pm
A question in this forum from Liudr - http://arduino.cc/forum/index.php/topic,50383.0.html - triggered me to implement a small runningAverage class. This can be found on - http://arduino.cc/playground/Main/RunningAverage .  A sample sketch shows how one can create running average on 2 different levels, - last minute, last hour - when one has a sample per second. In the sketch the sensor is simulated by the random function and samples are made far faster than one per second but in sample code this is allowed I guess :)

Please post comments and improvements in this thread,

thanks,
Rob

Code: [Select]
//
//    FILE: runningAverageHour.pde
//  AUTHOR: Rob Tillaart
//    DATE: 2012-12-30
//
// PUPROSE: show working of runningAverage per hour
//          in 2 steps - last minute + last hour
//          3 or more steps also possible
//

#include "RunningAverage.h"

RunningAverage raMinute(60);
RunningAverage raHour(60);

int samples = 0;

void setup(void)
{
  Serial.begin(115200);
  Serial.println("Demo RunningAverage lib - average per minute & hour");
  Serial.print("Version: ");
  Serial.println(RUNNINGAVERAGE_LIB_VERSION);
  raHour.clear();
  raMinute.clear();
}

void loop(void)
{
  long rn = random(0, 100);
  raMinute.addValue(rn);
  samples++;
 
  if (samples % 60 == 0) raHour.addValue(raMinute.getAverage());
 
  Serial.print("  raMinute: ");
  Serial.print(raMinute.getAverage(), 4);
  Serial.print("  raHour: ");
  Serial.println(raHour.getAverage(), 4);
}


updated example to reflect 0.2.02 version
Title: Re: RunningAverage Class on playground
Post by: liuzengqiang on Jan 31, 2011, 08:41 pm
Thanks a lot Rob. The sample code here is very nice. I love the two level average, nice for OOP. I guess I can set the sample amount to 60 or other quantity? I'll look into it later today!
Title: Re: RunningAverage Class on playground
Post by: graynomad on Feb 01, 2011, 04:05 am
Thanks for that, it'll save me writing something for an upcoming project. I'll play with it when the time comes.

_____
Rob
Title: Re: RunningAverage Class on playground
Post by: zaphod on Feb 28, 2011, 07:06 am

Rob, Just tried to get your example to go but comes up with a compile error.

\libraries\RunningAverage\RunningAverage.cpp:26: error: definition of implicitly-declared 'RunningAverage::~RunningAverage()'

Any help appreciated .

Thanks Rock.
Title: Re: RunningAverage Class on playground
Post by: robtillaart on Feb 28, 2011, 08:16 am
Which version of Arduino are you using?
- short term patch is to comment the ~RunningAverage() from .cpp and .h but that would create a memory leak. However as long as the object is in scope and there is no need to free the internal array that should not give a problem.
I'll have to dive in this one

------
update:

The .h file is missing a prototype of the destructor. Add    ~RunningAverage();   in the public section just under the constructor. That should fix it.
Playground article is updated.

Thanks for finding this one.
Rob
Title: Re: RunningAverage Class on playground
Post by: zaphod on Mar 01, 2011, 02:53 am
Rob, Thanks for that, the compiler gets past that, but now comes up with another problem,  error : request for member 'clr' in 'myRA', which is of non-class type 'RunningAverage ()()'.
This occurs also for member 'add' and 'avg' when I comment out the other problem lines.

I know zip about C++ so I'm at a loss .

Thanks Rock.
Title: Re: RunningAverage Class on playground
Post by: robtillaart on Mar 01, 2011, 09:14 am
Please post your sketch so I can check this one too.
Title: Re: RunningAverage Class on playground
Post by: zaphod on Mar 01, 2011, 11:45 pm
Rob, I found the problem , I had not deleted the .o file and restarted the IDE after changing the .h file.

compiles OK now.

Thanks for your help.

Rock.
Title: Re: RunningAverage Class on playground
Post by: robtillaart on Nov 20, 2012, 10:03 pm

Found a newer versio of the runningAverage class by Yuval Naveh here

- http://stromputer.googlecode.com/svn-history/r74/trunk/Arduino/Libraries/RunningAverage/RunningAverage.cpp -

most important addition:
Code: [Select]
void RunningAverage::trimToValue(float value)
{
clr();
add(value);
}


TODO: update the playground
Title: Re: RunningAverage Class on playground
Post by: robtillaart on Dec 30, 2012, 11:59 am
Finally found some time to update the runningAverage lib on the playground: - http://playground.arduino.cc//Main/RunningAverage -

changes:
1) There are some breaking changes in the method names to make them more descriptive;
// clr()  => clear()
// add(x) => addValue(x)
// avg() => getAverage()

2) new is the fillValue() function, based upon the trimValue() of Yuval Naveh  - see previous post
Code: [Select]
// fill the average with a value
// the param number determines how often value is added (weight)
// number should preferably be between 1 and size
void RunningAverage::fillValue(float value, int number)
{
clear();
for (int i = 0; i < number; i++)
{
addValue(value);
}
}

It fills the internal array with a number of identical values to get a starting value for average.
By adding more than one value, the initial average gets a certain weight.
This extends the original trimValue() so that's why I gave another name.

3) some small refactoring and added comments

as always comments, remarks and ideas are welcome
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 22, 2013, 11:47 pm
Hi there,

I´m using the runningAverage library and got some problems when clearing the averaging variabel every 300 counts.
Now, I did a Measurement series showing my raw sensor data and the avaregd data (see attached Differenzdruck_gemittelt_100.pdf). Looking at these curves, it is clearly visible that my avareged sensor data is going bogus when the averaging variabel is cleared after 300 samples. Now, looking at the library I found the option to use the a "fillvalue" instead. As the code is not that good commented, I just thought to ask what it actually does. However, Im loosing the peaks in my sensor data but seems to oscillate now. Well, to make a long story short, could someone explain the fillvalue feature of the running average library in more detail.

Code using the running average (plain and simple)
Code: [Select]

/*
*************************************
* Thread: THREAD_DifferentialPressure
* Get`s the differential pressure.
*************************************
*/

/* Global variables for this function */
float diff_Pressure = 0;
float offset_diff_pressure;              // offset which can be set by the user -> FUNC_OffsetDiffPressure (X_E_LCDML_MenuFunctions)
RunningAverage diff_p_RA(100);            // use default size for running average for differential pressure
int diff_p_samples = 0;                  // create a samples counter
 /* --------- INIT ---------
    * Initialization of this function
    * is only being done once at the start of the function
    */
void simpleThread_setup(THREAD_DifferentialPressure)
 {  
 /* Init Function */
 diff_p_RA.clear();                     // Running Average for diff. pressure; explicitly start clean
 }
   
 /* --------- LOOP ----------
    * This is the place for code which is going to be repeated constantly
    * e.g. a time-diplay or the like.
    */
boolean simpleThread_loop(THREAD_DifferentialPressure)
 {
 /* Loop Function */
 diff_p_RA.addValue(getDifferentialPressure()); // read out the sensor value and add to averaging variable
 diff_p_samples++;                      // increase samples counter
 // set averaged differential pressure  - pressure offset set by the user
 diff_Pressure = diff_p_RA.getAverage() - offset_diff_pressure;
 Serial.println(diff_Pressure);
 if (diff_p_samples == 300)             // handle large samplenumbers...
 {
   diff_p_samples = 0;                // ...delete samples counter, and...
   diff_p_RA.clear();                 // ...clear the averaging variable.
 }
 else if (isnan(diff_Pressure) || diff_Pressure < 0.00)
 {
   diff_Pressure = 0.00;    // set diff_Pressure to zero if the reading is NaN - Not a Number, or lower than zero
 }
 return true;
 }


Same code using the running average wih the fill value option:
Code: [Select]

/*
*************************************
* Thread: THREAD_DifferentialPressure
* Get`s the differential pressure.
*************************************
*/

/* Global variables for this function */
float diff_Pressure = 0;
float offset_diff_pressure;              // offset which can be set by the user -> FUNC_OffsetDiffPressure (X_E_LCDML_MenuFunctions)
RunningAverage diff_p_RA(100);            // use default size for running average for differential pressure
int diff_p_samples = 0;                  // create a samples counter
 /* --------- INIT ---------
    * Initialization of this function
    * is only being done once at the start of the function
    */
void simpleThread_setup(THREAD_DifferentialPressure)
 {  
 /* Init Function */
 diff_p_RA.fillValue(0,50);                     // Running Average for diff. pressure; explicitly start clean
 }
   
 /* --------- LOOP ----------
    * This is the place for code which is going to be repeated constantly
    * e.g. a time-diplay or the like.
    */
boolean simpleThread_loop(THREAD_DifferentialPressure)
 {
 /* Loop Function */
 diff_p_RA.addValue(getDifferentialPressure()); // read out the sensor value and add to averaging variable
 diff_p_samples++;                      // increase samples counter
 // set averaged differential pressure  - pressure offset set by the user
 diff_Pressure = diff_p_RA.getAverage() - offset_diff_pressure;
 if (diff_p_samples == 300)             // handle large samplenumbers...
 {
   diff_p_samples = 0;                  // ...delete samples counter, and...
   diff_p_RA.fillValue(diff_Pressure, 25); // ...clear the averaging variable.
 }
 else if (isnan(diff_Pressure) || diff_Pressure < 0.00)
 {
   diff_Pressure = 0.00;    // set diff_Pressure to zero if the reading is NaN - Not a Number, or lower than zero
 }
 return true;
 }



Thank you for any help.
Best regards,
Jan
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 27, 2013, 12:54 pm
ANSWER BY ROB TILLAART (send by e-mail):

In the example (of the RunningAverage library) I am clearing the RA variable every 300 count so people see how it can be done and what its effect is on the RA. And that effect is exactly what you see in the RA, the smooth line breaks. The number 300 is just arbitrary.

There can be several reasons why one wants to clear the RA var.
(1) one starts a (really) new series of measurements, Suppose one makes a RA measurement of the light-intensity for one minute once per hour. Then you really want to start clean to make the measurements independent.
(2) when you switch to another range of your input , e.g. you switch the analogReference of a VoltMeter sketch from 5.0 to 1.1Volt.
(3) when you switch the timing between the values, if you have a series with a 1 second interval and you change the interval to 1 minute you must clear the RA otherwise you will compare apples with pears.
(4) when the value measured changes more than a certain value (jumps to another level) one might want to follow immediately.
In the PDF's on the forum post (see link above) there is such a jump around 500 and 900.

That said, in most sketches that run continuously do not need to clear/reset the RA. it can go on and on and on forever.

On the forum there is also the question about fillValue(value, number);
This function can be used to initialize the RA on a certain value with a certain weight,
Example (assume buffersize = 10
if you do a clear() followed by a addValue(100); the RA will be 100. another addValue(50) will set the RA to 75.0.
if you do a fillValue(100, 10), followed by a addValue(100); the RA will be 100, another addValue(50) will set the RA to 95.0,
if you do a fillValue(100, 5), followed by a addValue(100); the RA will be 100, another addValue(50) will set the RA to 92,8571...,

So fillValue has its use especially at the start of a measurement
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 27, 2013, 01:05 pm
Hi Rob,

thanks for the help! Another thing I was stumbling upon was the question if there were any possible chances of creating an RA-array:
Code: [Select]

RunningAverage diff_p_RA(50);

in a way that the size (int) would be user configurable during runtime? Now, as I understand it, there is quite a fuz about growing or reducing an array during runtime - meaning that it is not that simple. However, maybe you have an idea for an approach one could try implementing. Maybe, using a fixed large array of the maximum expected size (200 would be a reasonable value -I guess) and then only using parts of that array - as the user defines the averaging to be done between 0 and 200 values.

Best regards,
Jan
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 27, 2013, 05:10 pm
Hi Rob,

I did some more measurements and used RunningMedian instead of RunningAverage. The Code I used can be found here:
Code: [Select]


/*
*************************************
* Thread: THREAD_DifferentialPressure
* Get`s the differential pressure.
*************************************
*/

/* Global variables for this function */
float diff_Pressure_median = 0;
float diff_Pressure = 0;
float offset_diff_pressure;     // offset which can be set by the user -> FUNC_OffsetDiffPressure (X_E_LCDML_MenuFunctions)
int n_averaging;                // variable can be set by the user to define ow many measurements shall be averaged
RunningMedian diff_p_RA(200);   // use a uer defined size for the running averaging of the differential pressure
 /* --------- INIT ---------
    * Initialization of this function
    * is only being done once at the start of the function
    */
void simpleThread_setup(THREAD_DifferentialPressure)
 {  
 /* Init Function */
 //diff_p_RA.clear;              // Running Average for diff. pressure; explicitly start clean
 }
   
 /* --------- LOOP ----------
    * This is the place for code which is going to be repeated constantly
    * e.g. a time-diplay or the like.
    */
boolean simpleThread_loop(THREAD_DifferentialPressure)
 {
 /* Loop Function */
 diff_p_RA.add(getDifferentialPressure()); // read out the sensor value and add to averaging variable
 // set averaged differential pressure  - pressure offset set by the user
 diff_Pressure = diff_p_RA.getAverage() - offset_diff_pressure;
 diff_Pressure_median = diff_p_RA.getMedian() - offset_diff_pressure;
 Serial.print(diff_Pressure);
 Serial.print(";");
 Serial.println(diff_Pressure_median);
 if (isnan(diff_Pressure) || diff_Pressure < 0.00)
 {
   diff_Pressure = 0.00;    // set diff_Pressure to zero if the reading is NaN - Not a Number, or lower than zero
 }
 return true;
 }

The results of these measurements can be found in the attached pdf file "Diff_Pressure_averaged_median.pdf". Now there is something strange about the averaging value. The buffer size seems not to change.  even the 200 value buffer size ("RunningMedian diff_p_RA(200); ") looks more like the 25 or even 10. The RunningMedian Code I used is from your (Rob Tillaart) github repository. Furthermore, I expected something else from the RunningMedian. Let me know what you think.

Best regards,
Jan

P.S.: I now added the raw data of these measurements.
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 27, 2013, 06:09 pm
Hi there,

just to complete the statements above, I made another series of measurements (see "Diff_Pressure_RunningAverage.pdf") using the same buffer sizes. But now I used RunningAverage again. As can be clearly seen, the output is as expected. I did not clear the variable as this is not necessary with these measurements.
so one may draw the conclusion  that there is something wrong with the code of the RunningMedian Library.

Here is the code sniplet where I calculate my RunningAverage for the differential pressure:
Code: [Select]

/*
*************************************
* Thread: THREAD_DifferentialPressure
* Get`s the differential pressure.
*************************************
*/

/* Global variables for this function */
float diff_Pressure_median = 0;
float diff_Pressure = 0;
float offset_diff_pressure;     // offset which can be set by the user -> FUNC_OffsetDiffPressure (X_E_LCDML_MenuFunctions)
int n_averaging;                // variable can be set by the user to define ow many measurements shall be averaged
RunningAverage diff_p_RA(200);   // use a uer defined size for the running averaging of the differential pressure
 /* --------- INIT ---------
    * Initialization of this function
    * is only being done once at the start of the function
    */
void simpleThread_setup(THREAD_DifferentialPressure)
 {  
 /* Init Function */
 diff_p_RA.clear();              // Running Average for diff. pressure; explicitly start clean
 }
   
 /* --------- LOOP ----------
    * This is the place for code which is going to be repeated constantly
    * e.g. a time-diplay or the like.
    */
boolean simpleThread_loop(THREAD_DifferentialPressure)
 {
 /* Loop Function */
 diff_p_RA.addValue(getDifferentialPressure()); // read out the sensor value and add to averaging variable
 // set averaged differential pressure  - pressure offset set by the user
 diff_Pressure = diff_p_RA.getAverage() - offset_diff_pressure;
 Serial.println(diff_Pressure);
 if (isnan(diff_Pressure) || diff_Pressure < 0.00)
 {
   diff_Pressure = 0.00;    // set diff_Pressure to zero if the reading is NaN - Not a Number, or lower than zero
 }
 return true;
 }


Best regards,
Jan

P.S.: I now added the raw data of these measurements.
Title: Re: RunningAverage Class on playground (updated)
Post by: ivanseidel on Oct 28, 2013, 05:04 am
Take a look at my Library, that works perfecly for a moving average...

https://github.com/ivanseidel/Gaussian
Title: Re: RunningAverage Class on playground (updated)
Post by: Sembazuru on Oct 28, 2013, 08:04 am

[... snipped ...]
The results of these measurements can be found in the attached pdf file "Diff_Pressure_averaged_median.pdf". Now there is something strange about the averaging value. The buffer size seems not to change.  even the 200 value buffer size ("RunningMedian diff_p_RA(200); ") looks more like the 25 or even 10. The RunningMedian Code I used is from your (Rob Tillaart) github repository. Furthermore, I expected something else from the RunningMedian. Let me know what you think.

Best regards,
Jan

P.S.: I now added the raw data of these measurements.



If you look in RunningMedian.h you will see
Code: [Select]

#define MEDIAN_MAX_SIZE     19          // adjust if needed


If you don't change that, requesting a median array size of anything larger than 19 will only give you a 19 element array to run your medians and averages against. Did you change that line before compiling your sketch that requests a 200 element array?
Title: Re: RunningAverage Class on playground (updated)
Post by: jabami on Oct 28, 2013, 11:45 am
Hi there,

thank you  Sembazuru for pointing that out.  I guess that means I should have set "MEDIAN_MAX_SIZE" to somewhat around 200 and then define a
Code: [Select]
RunningMedian samples = RunningMedian(50); Well, as I learned yesterday the RunningMedian takes more processing time as it needs to sort the array. As I´m taking my measurements each 50ms I guess that would not be an option - especially not with a large array size (which seem absolutley logical now to limit the array size). I´m now using the RunningAverage-library as I found my optimum array size for my setup via the tests I did. So everything is fine now. But I still wonder if the array size can be made user configurable during runtime.

@ivanseidel: Thank you for pointing me to your library. But I allready implemented the RunningAverage-library by Rob Tillaart and only needed some fine tuning (and understanding on whats going on behind the scenes). The last results I got were  suficiently constant. I just needed to adjust the array size to get the timing right (because a large array size means smoother data but more timeleg  on steep gradients ).

Best regards,
Jan 
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Oct 28, 2013, 01:21 pm
I have tried to reproduce the getMedian() error but I could not. Using the [partial] data from your zip file I got no errors.

A possible cause I cannot verify is how much free RAM you have as it might be a problem during allocating the internal arrays as the lib uses dynamic memory. That could explain the behaviour seen.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Oct 28, 2013, 01:25 pm
Quote
But I still wonder if the array size can be made user configurable during runtime.

The lib could be rewritten to use dynamic memory and you could reallocate the memory. However as an Arduino has very limited RAM this gives fragmented (useless) memory quite fast.

that said, you might try with largest allocation first and gradually decrease the size to an optimum. Then the reallocation should less fragment the memory in theory ...


Title: Re: RunningAverage Class on playground (updated)
Post by: ivanseidel on Oct 28, 2013, 02:28 pm

Quote
But I still wonder if the array size can be made user configurable during runtime.

The lib could be rewritten to use dynamic memory and you could reallocate the memory. However as an Arduino has very limited RAM this gives fragmented (useless) memory quite fast.

that said, you might try with largest allocation first and gradually decrease the size to an optimum. Then the reallocation should less fragment the memory in theory ...



That's true, but it's already implemented with my library (I use a self made LinkedList). Worth a try =)

You can change the size whenever you want, and it will not crash the code.

Also, you can use it with basic numbers 10 = Gaussian(10); or with numbers with variance: Gaussian(10,variance);, witch gives you much more flexibility on the average...

Just a tip... take a look at the documentation.

https://github.com/ivanseidel/Gaussian
Title: Re: RunningAverage Class on playground (updated)
Post by: eXistenZ on May 07, 2014, 07:18 pm
What is the easiest way to modify RunningAverage Class to calculate 4 average variables simultaneously?

:smiley-eek-blue:
Title: Re: RunningAverage Class on playground (updated)
Post by: ivanseidel on May 07, 2014, 08:01 pm
Code: [Select]

GaussianAverage avg1(NUM_AVGS);
GaussianAverage avg2(NUM_AVGS);
GaussianAverage avg3(NUM_AVGS);
GaussianAverage avg4(NUM_AVGS);

//In code to add values to the moving average
avg1 += 10;
avg2 += xx...;

// Then process and get the result
float reault1 = avg1.process().mean;


Library: http://github.com/ivanseidel/Gaussian
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on May 07, 2014, 08:07 pm
Create multiple separate objects

Code: (not tested) [Select]
//
//    FILE: runningAverageTest.ino
//  AUTHOR: Rob Tillaart
//    DATE: 2014-05-07
//
// PUPROSE: show working of runningAverage
//

#include "RunningAverage.h"

RunningAverage RA0(15);
RunningAverage RA1(13);
RunningAverage RA2(11);
RunningAverage RA3(9);
RunningAverage RA4(7);
RunningAverage RA5(5);

int samples = 0;

void setup(void)
{
 Serial.begin(115200);
 Serial.println("Demo RunningAverage lib");
 Serial.print("Version: ");
 Serial.println(RUNNINGAVERAGE_LIB_VERSION);

 clearAll();
}

void loop(void)
{
 RA0.addValue(analogRead(A0));
 RA1.addValue(analogRead(A1));
 RA2.addValue(analogRead(A2));
 RA3.addValue(analogRead(A3));
 RA4.addValue(analogRead(A4));
 RA5.addValue(analogRead(A5));

 Serial.print(RA0.getAverage(), 3);
 Serial.print(",");
 Serial.print(RA1.getAverage(), 3);
 Serial.print(",");
 Serial.print(RA2.getAverage(), 3);
 Serial.print(",");
 Serial.print(RA3.getAverage(), 3);
 Serial.print(",");
 Serial.print(RA4.getAverage(), 3);
 Serial.print(",");
 Serial.println(RA5.getAverage(), 3);

 samples++;

 if (samples == 300)
 {
   samples = 0;
   clearAll();
 }
 delay(100);
}

void clearAll()
{
 RA0.clear();  
 RA1.clear();  
 RA2.clear();  
 RA3.clear();  
 RA4.clear();  
 RA5.clear();  
}

running this program should generate output you can graph in a spreadsheet giving insight in the effects of the internal buffer.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on May 07, 2014, 08:22 pm

Or use an array of objects.  Note the allocation is somewhat different

(note not tested, highly experimental)

Code: [Select]
//
//    FILE: runningAverageTest.ino
//  AUTHOR: Rob Tillaart
//    DATE: 2014-05-07
//
// PUPROSE: show working of runningAverage
//

#include "RunningAverage.h"

RunningAverage *RA[6];  // 6 pointers to RA objects
int pin[6] = {
  A0, A1, A2, A3, A4, A5 };
int size[6] = { 15, 13, 11, 9, 7, 5 };

int samples = 0;

void setup(void)
{
  Serial.begin(115200);
  Serial.println("Demo RunningAverage lib");
  Serial.print("Version: ");
  Serial.println(RUNNINGAVERAGE_LIB_VERSION);

  for (int i=0;i<6; i++)
  {
    RA[i] = new RunningAverage( size[i] );
    RA[i]->clear();
  }
}

void loop(void)
{
  for (int i=0; i<6; i++)
  {
    RA[i]->addValue(analogRead(pin[i]));
  }

  for (int i=0; i<6; i++)
  {
    Serial.print(RA[i]->getAverage(), 3);
    Serial.print(",");
  }
  Serial.println();

  samples++;
  if (samples == 300)
  {
    samples = 0;
    for (int i=0;i<6; i++)
    {
      RA[i]->clear();
    }
  }
  delay(100);
}
Title: Re: RunningAverage Class on playground (updated)
Post by: Chrismolloy on Jul 02, 2014, 07:37 am
I am having overflow errors in visual micro debugger in ms visual studio.

I have 15 instances of RunningAverage sized from 24 to 60.

I am getting random ovf errors on calls to getAverage().

S = savg.getAverage() generates an ovf error at random times.
Title: Re: RunningAverage Class on playground (updated)
Post by: Chrismolloy on Jul 02, 2014, 02:02 pm

I am having overflow errors in visual micro debugger in ms visual studio.

I have 15 instances of RunningAverage sized from 24 to 60.

I am getting random ovf errors on calls to getAverage().

S = savg.getAverage() generates an ovf error at random times.


here is the code

Code: [Select]

/*
This code creates a schedule for events every second, minute & hour
*/

#include <Metro.h> //Include Metro library

// Create metro objects and set intervals to one second, minute & hour
Metro Second = Metro(1000); //one second
Metro Minute = Metro(2000); //one minute
Metro Hour = Metro(4000); //one hour

#include "RunningAverage.h"

//Create running average objects
RunningAverage Sensor1Seconds(60);
RunningAverage Sensor1Minutes(60);
RunningAverage Sensor1Hours(24);
RunningAverage Sensor2Seconds(60);
RunningAverage Sensor2Minutes(60);
RunningAverage Sensor2Hours(24);
RunningAverage Sensor3Seconds(60);
RunningAverage Sensor3Minutes(60);
RunningAverage Sensor3Hours(24);
RunningAverage Sensor4Seconds(60);
RunningAverage Sensor4Minutes(60);
RunningAverage Sensor4Hours(24);
RunningAverage Sensor5Seconds(60);
RunningAverage Sensor5Minutes(60);
RunningAverage Sensor5Hours(24);


//variables to track time elapsed
int seconds = 0;
int minutes = 0;
int hours = 0;
int led = 13;

void blink()
{
digitalWrite(led,1);
int m = millis();
while(millis() < m+500);
//for(int n=1;n<100000;n=++n);
digitalWrite(led,0);
}

void setup()
{
   pinMode(led,OUTPUT);
blink();
Sensor1Seconds.clear();
Sensor1Minutes.clear();
Sensor1Hours.clear();
Sensor2Seconds.clear();
Sensor2Minutes.clear();
Sensor2Hours.clear();
Sensor3Seconds.clear();
Sensor3Minutes.clear();
Sensor3Hours.clear();
Sensor4Seconds.clear();
Sensor4Minutes.clear();
Sensor4Hours.clear();
Sensor5Seconds.clear();
Sensor5Minutes.clear();
Sensor5Hours.clear();
}

float s1=0;
float s2=0;
float s3=0;
float s4=0;
float s5=0;
float m1=0;
float m2=0;
float m3=0;
float m4=0;
float m5=0;
float h1=0;
float h2=0;
float h3=0;
float h4=0;
float h5=0;
float rn;

void SensorSecondUpdate() {
rn = random(0, 1000);
       Sensor1Seconds.addValue(rn);
s1 = Sensor1Seconds.getAverage();
rn = random(0, 1000);
       Sensor2Seconds.addValue(rn);
s2 = Sensor2Seconds.getAverage();
rn = random(0, 1000);
       Sensor3Seconds.addValue(rn);
s3 = Sensor3Seconds.getAverage();
rn = random(0, 1000);
       Sensor4Seconds.addValue(rn);
s4 = Sensor4Seconds.getAverage();
rn = random(0, 1000);
       Sensor5Seconds.addValue(rn);
s5 = Sensor5Seconds.getAverage();

}

void SensorMinuteUpdate() {
float x = Sensor1Seconds.getAverage();
Sensor1Minutes.addValue(x);
m1 = Sensor1Minutes.getAverage();
Sensor1Seconds.clear();
x = Sensor2Seconds.getAverage();
Sensor2Minutes.addValue(x);
m2 = Sensor2Minutes.getAverage();
Sensor2Seconds.clear();
x=Sensor3Seconds.getAverage();
Sensor3Minutes.addValue(x);
m3 = Sensor3Minutes.getAverage();
Sensor3Seconds.clear();
x = Sensor4Seconds.getAverage();
Sensor4Minutes.addValue(x);
m4 = Sensor4Minutes.getAverage();
Sensor4Seconds.clear();
x=Sensor5Seconds.getAverage();
Sensor5Minutes.addValue(x);
m5 = Sensor5Minutes.getAverage();
Sensor5Seconds.clear();
}

void SensorHourUpdate() {
float x=Sensor1Minutes.getAverage();
Sensor1Hours.addValue(x);
h1 = Sensor1Hours.getAverage();
x=Sensor2Minutes.getAverage();
Sensor2Hours.addValue(x);
h2 = Sensor2Hours.getAverage();
x=Sensor3Minutes.getAverage();
Sensor3Hours.addValue(x);
h3 = Sensor3Hours.getAverage();
x=Sensor4Minutes.getAverage();
Sensor4Hours.addValue(x);
h4 = Sensor4Hours.getAverage();
x=Sensor5Minutes.getAverage();
Sensor5Hours.addValue(x);
h5 = Sensor5Hours.getAverage();
}


void loop()
{

 if (Second.check() == 1) { // check if its been a second
seconds = ++seconds;
SensorSecondUpdate();
blink();
//put stuff here to do every second
 }

   if (Minute.check() == 1) { // check if its been a minute
minutes = ++minutes;
SensorMinuteUpdate();
//put stuff here to do every minute
 }

if (Hour.check() == 1) { // check if its been an hour
hours = ++hours;
SensorHourUpdate();

//put stuff here to do every hour
 }

}

moderatore: added code tags - # button above smileys
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Jul 02, 2014, 07:13 pm
you have 10 RA objects of size 60 and 5 RA objects of size 24, that is a lot of memory (at least for an UNO)

Every object has 10 bytes local vars and 4 bytes per element.

15 objects = 150 bytes
10 x 60 x 4 = 2400 bytes
5 x 24 x 4 = 480 bytes
--------------------
roughly 3000+ bytes of SRAM

What board are you using?
If it is an UNO, it does not have that much memory, ==> Get a MEGA.

Learning point for me: The RA class has no error handling or state to see its "health".
Analysis:
The class does not check if the allocation of the needed "arrays" succeed, but the code behaves as if it does so.
That means that only the first RA object are healthy and the others have internal pointers pointing to (probably) NULL.

To make it more robust the interface of the Class could change to have a method
Code: [Select]
bool begin(uint8_t size) ;
that returns true is size elements could be allocated.
The allocation should be removed from the constructor.
// this is a breaking interface change

Another alternative is to have a method
Code: [Select]
uint8_t internalSize();
That returns the size of the internal size. This returns zero if allocation had failed and the internal size otherwise.
// this would not break the existing interface and would increase footprint not much.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Jul 03, 2014, 10:18 pm
Updated the library to version 0.2.04 ( 0.2.03 was skipped - dev only version)

To solve the problem of Chrismolloy above I added code to check if the internal array could be allocated. The previous 0.2.02 version was really opportunistic ;)
If the array cannot be allocated the size of the array is set to 0, the new method getSize() can be used to check if the allocation worked. Having an internal size of zero the Class cannot accept new values and the average will be NAN as it cannot be calculated.

The size of the lib increased about 20 bytes, which is imho acceptable for this "safety net".

As always remarks and comments are welcome.

- http://playground.arduino.cc/Main/RunningAverage -
- https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage -
Title: Re: RunningAverage Class on playground (updated)
Post by: GPSJay on Dec 16, 2014, 03:42 pm
Rob,

I've been using your library, in conjunction with TinyGPS++ library, in a GPS routine to average the GPS position and help reduce the position wander, and had some interesting results. Whilst the longitude seems to function correctly e.g. 1.01834380, the latitude seems to stop or stall and not continue example: 51.34103012.

Note that I'm going to 8 decimal points for the LAT/ LON position. I wondered if there is a limitation in the RA library that limits the length or size of the number?

Also, would it be capable of taking both + and - numbers?

Comments or help very much appreciated!

Regards.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Dec 16, 2014, 08:53 pm
The RA lib uses floats which are on an AVR based Arduino 32bits IEEE754 floats. This means a 23 bit mantissa which equals about 7 significant digits.

If you are using an ARM based Arduino, which supports 64 bit double you could patch the library by replacing all float types with double. Should be a good change anyway. On my todo list.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Dec 16, 2014, 10:10 pm
updated Running Average Library to version 0.2.05 on GITHUB
- https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage -

changed float to double
==> support for double on ARM of the Arduino DUE ...
Title: Re: RunningAverage Class on playground (updated)
Post by: GPSJay on Dec 18, 2014, 03:00 pm
Thanks Rob.

BTW I'm using a MEGA with GPS engines running at 4Hz...
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Mar 07, 2015, 09:28 pm
updated Running Average Library to version 0.2.06 on GITHUB

- https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage -

+ minor edits
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Apr 10, 2015, 09:29 pm
skip version 0.2.07 (contained typos)

updated Running Average Library to version 0.2.08 on GITHUB
- https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage -

since 0.2.06
+ added getMin() and getMax() - thanks to Eric M.
+ refactored

Note getMin() and getMax() gives the minimum/maximum since last call to clear(),
==> so it is not the min/max of the internal buffer.
Title: Re: RunningAverage Class on playground (updated)
Post by: Sembazuru on Apr 20, 2015, 03:38 am
Do you have any plans on getting your libraries added to the 1.6.2+ library manager? I imagine it would take quite a bit of work because it seems at first brush that each library should be it's own repository (instead of how you have several libraries in one repository).

I think AdaFruit has published a script to help mass publishing of libraries.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Apr 20, 2015, 09:56 pm
No I do not have such plans in near future as my time is quite limited.

And I am using 1.5.8 which works quite well for me.
Title: Re: RunningAverage Class on playground (updated)
Post by: Sembazuru on Apr 22, 2015, 01:42 am
No I do not have such plans in near future as my time is quite limited.

And I am using 1.5.8 which works quite well for me.
OK. Fair answer.
Title: Re: RunningAverage Class on playground (updated)
Post by: aaronblanchard on Aug 31, 2015, 07:17 am
One thing to be wary of when using this library is floating point imprecision.

I left a unit running for a week sampling 10 times a second and was very confused when I came back as to why my average (over the last 50 samples) was very different to the actual current input.

Looking into the code, the RunningAverage::addValue function subtracts a Double and then adds a Double.  Adding and subtracting of Doubles (and Floats) can lead to a huge degree of imprecision. 

I'm not sure what the best solution is here, calculating the sum total every time getAverage() is called is the obvious one, but that leads to somewhat more function overhead.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Aug 31, 2015, 09:51 pm
You are completely right. Thanks for this observation.

10 samples/second = 864.000 /day = 6 million per week.

The number of samples is approaching the accuracy of the (IEEE754) float (7 digits) when one reaches 1 million samples (1 day in your case) . This means if the average sample has 3 digits, one will loose 2 digits every time.

The "trick" of remove one double and add one double is an efficient way to calculate the sum of the array of samples. And yes when using this trick the error will add up in sum. In theory after 100 samples the last two digits are suspected to be incorrect. Need to investigate the effect.

Solution is to (sort and) add all elements of the array every time, giving a constant error. Consequence is a slower execution. The sorting will improve the sum when the dynamic range of the numbers is large.

A practical solution for the library might be to do a proper sum e.g. every 100 or 1000 times a value is added (new parameter?). The lower this number the more accurate the sum. The price is a performance penalty once every 100 / 1000 times. There are several variations possible.
A good value for the number will also depend on the size of the numbers added. Ideally one would like to have a sort of error counter that makes an estimate of the max error and when a threshold is reached do a recalculation of sum.

The easiest solution is to redo the function Average() and rename the current one to fastAverage();

in short food for thought.

thanks




Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Aug 31, 2015, 10:02 pm

quick patch:
Code: (snippet for runningAverage.cpp) [Select]

// returns the average of the data-set added sofar
double RunningAverage::getAverage() const
{
    if (_cnt == 0) return NAN;
    double sum = 0;
    for (uint8_t i = 0; i < _cnt; i++)
    {
        sum += _ar[i];
    }
    return sum / _cnt;
}

double RunningAverage::getFastAverage() const
{
    if (_cnt == 0) return NAN;
    return _sum / _cnt;
}

the .h file must add a signature for

double getFastAverage() const;
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Sep 05, 2015, 12:24 pm
updated Running Average Library to version 0.2.11 on GITHUB
https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage (https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage) -

The following changes were made since 0.2.08
(request aaronblanchard)
+ getAverage() renamed to getFastAverage() as it is fast but less accurate.
+ reimplemented getAverage() to be accurate, but a bit slower.
getAverage() sums all elements in the internal buffer and average them. This solves the remark from  aaronblanchard a few posts ago that the fastAverage() drifts from the real value. An example sketch is included to show this drift.

(request by mail)
+ added GetMinInBuffer() to make difference with getMin()
+ added GetMaxInBuffer() to make difference with getMin()
Note getMin() and getMax() gives the minimum/maximum since last call to clear(). An example sketch shows the differences between the methods.

+ refactored

snippet from drift test (included as example)
Code: [Select]

COUNT       AVG         FASTAVG     DIFF        MAXDIFF sofar
30750000 0.5153000 0.5152977 0.0000023 0.0000522
30760000 0.5241000 0.5240974 0.0000026 0.0000522

The sample sketch adds values between 0.0 and 0.999 in a loop. The maximum difference after 30 million++ additions is relative still low. There will be input streams which are more sensitive for the fastAverage algorithm. So when max accuracy is a requirement please use getAverage().

as always remarks and bug reports are welcome,
Rob
Title: Re: RunningAverage Class on playground (updated)
Post by: Leonas on Jan 30, 2016, 12:24 pm
updated Running Average Library to version 0.2.11 on GITHUB
https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage (https://github.com/RobTillaart/Arduino/tree/master/libraries/RunningAverage) -
Maybe a stupid question but where is the download button for this library? I must be missing something.   :-[ 
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Jan 30, 2016, 02:18 pm
No stupid question, the [download zip] button is two levels up (right side)

https://github.com/RobTillaart/Arduino (https://github.com/RobTillaart/Arduino)

or use

https://github.com/RobTillaart/Arduino/archive/master.zip
Title: Re: RunningAverage Class on playground (updated)
Post by: Leonas on Jan 31, 2016, 04:48 pm
No stupid question, the [download zip] button is two levels up (right side)

https://github.com/RobTillaart/Arduino (https://github.com/RobTillaart/Arduino)
OK, thanks, sorry that I missed that. I was too deep. ;)
Title: Re: RunningAverage Class on playground (updated)
Post by: odometer on Feb 07, 2016, 08:53 pm
There are tricks you can use to improve accuracy when adding up a lot of floats.
For example, you can use a long to hold the "whole" part of the sum, and a float to hold the fractional part.
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Feb 09, 2016, 09:52 pm
Good point, it's a variation of - https://en.wikipedia.org/wiki/Kahan_summation_algorithm

that said accuracy tricks often come at a price of footprint, performance or sometimes the working range of the code. The more you know about the data to process the easier/better you can tune an algorithm for accuracy/performance etc.
Title: Re: RunningAverage Class on playground (updated)
Post by: jano on Mar 16, 2016, 04:16 am
hi!

First: Thanks for publishing your code, great work!

I'm using it to filter the input of opto sensors for reading the speed of two dc motors, but since the speed of the motors is variable the cutoff of the filter should too, exist a way to change the amount of samples in the running average on the fly?
Title: Re: RunningAverage Class on playground (updated)
Post by: lucy on Jul 22, 2017, 10:05 pm
Can I run 3 copies of RunningAverage.h at the same time? I am attempting to find relative accelerometer motion in three axis simultaneously to produce a z score. this is a test using example furnished with the library. Can I interleave or multiplex. Sample size will probably need tobe altered on thefly as code progresses. Thanks a lot.


//
//    FILE: runningAverageTest.pde
//  AUTHOR: Rob Tillaart
// VERSION: 0.1.01
//    DATE: 2012-12-30
//
// PUPROSE: show working of runningAverage
//

#include "RunningAverage.h"

RunningAverage myRA(10);
int samples = 0;

void setup(void)
{
  Serial.begin(115200);
  Serial.println("Demo RunningAverage lib");
  Serial.print("Version: ");
  Serial.println(RUNNINGAVERAGE_LIB_VERSION);
  myRA.clear(); // explicitly start clean
}

void loop(void)
{ int x=analogRead(A0);
  int u=myRA.getAverage();
  int s=myRA.GetStandardDeviation();
 
  int z=(x-u)/s;
  //long rn = random(0, 1000);
  myRA.addValue(x);
 // samples++;
 // Serial.print(samples);
 // Serial.print("\t Running Average: ");
  Serial.println(z);
 
 /* if (samples == 300)
  {
    samples = 0;
    myRA.clear();
    Serial.println();
  }*/
  delay(10);
}
Title: Re: RunningAverage Class on playground (updated)
Post by: robtillaart on Jul 24, 2017, 02:19 pm
Yes you can by allocating 3 running Average objects.

Add the X values to the RA_X object, the Y values to the RA_Y object etc

see small sketch below

Code: (not tested) [Select]
//
//    FILE: runningAverage.ino
//  AUTHOR: Rob Tillaart
// VERSION: 0.0.1
//    DATE: 2017-07-24
// PURPOSE: demo
//
// HISTORY:
// 0.0.1 - 2017-07-24 initial version

#include "RunningAverage.h"

RunningAverage RA_X(10);
RunningAverage RA_Y(10);
RunningAverage RA_Z(10);

void setup(void)
{
  Serial.begin(115200);
  Serial.println("Demo RunningAverage lib");
  Serial.print("Version: ");
  Serial.println(RUNNINGAVERAGE_LIB_VERSION);

  RA_X.clear();
  RA_Y.clear();
  RA_Z.clear();
}

void loop(void)
{
  int x = analogRead(A0);
  int y = analogRead(A1);
  int z = analogRead(A2);

  RA_X.addValue(x);
  RA_Y.addValue(y);
  RA_Z.addValue(z);

  Serial.print("RA_X: \t");
  Serial.println(RA_X.getAverage());
  Serial.println(RA_X.GetStandardDeviation());

  // idem for RA_Y and RA_Z

  int u = RA_X.getAverage();
  int s = RA_X.GetStandardDeviation();
  int zzz = (x - u) / s;

  Serial.print("ZZZ:\t");
  Serial.println(zzz);

  delay(10);
}
Title: Re: RunningAverage Class on playground (updated)
Post by: Mikusban on Oct 06, 2018, 10:24 pm
Is it possible to only clear Min/Max values?