Sampling and processing data

Hi to all gurus of the Forum,

My little project get data from a diesel engine, through various sensors, and now, I need to process them.

Maybe the smoothing and sampling commands already exist in some library.

What I need is:

a) to sample, say, a variable over a period of time, and calculate the mean of the variable for that period.

Could be one value each second, for example, and calculate mean value in 10 seconds, or over a minute.

Is there a function, a commmand or a library for this?

b) a function that can calculate a moving average over a sample of values for a variable?

c) a way of storing values in memory, as an array, for example?

Thank you all in advance.

OldBeaver

AverageList:
http://www.arduino.cc/playground/Code/AverageList
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1242877176

Array:
http://www.arduino.cc/playground/Code/Array
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1244857483#2¨

You can use these functions:

int  size()
type getMax()
type getMin()
type getAverage()

type corresponds to the type you initialized the Array object with.

For instance, if you’ll want to log bytes:

const byte SIZE = 10;
byte rawArray[SIZE] = {1,2,3,4,5,6,7,8,9,10};
Array<byte> array = Array<byte>(rawArray,SIZE);

getMax, getMin and getAverage will return a byte.

These libraries turned out to be almost identical. Maybe I’ll have to deprecate one.

Hi,

This is a small sketch that intents to test a routine for sampling n data inputs (in this sketch n=25), promediate its values, display them or use in further calculations.

It includes the function LcdPrintFloat developed by mem, and kindly shared with me.

// ============ Test Routine for mean calculation over n obs ========
//
// This routine computes mean of 25 samples of variable of interest.

#include <LiquidCrystal.h>

//create object to control an LCD.
LiquidCrystal lcd(12, 11, 6, 7, 8, 9, 10);

int valx; // the variable of interest
int y; // sum values
int nrox; // counts obs
float x; // aux variable

void setup() {
pinMode(13, OUTPUT); //LED for debugging
y = 0; // initializes sum
}

void loop() {
digitalWrite(13, HIGH); //light the debug LED

// ============ generates random numbers ======
valx = random(1,100);
y = y + valx; // sums values to promediate
delay (300);

// ======= cuenta ciclos =======
nrox = nrox +1; // keeps count of nr of obs.

// ======= prints values =======

lcd.setCursor(0,0);
lcd.print (“x=”); lcd.print(valx); lcd.print(" “);
lcd.print (“y=”); lcd.print (y); lcd.print (” “);
lcd.print (“ob=”); lcd.print (nrox); lcd.print (” ");

lcd.setCursor(0,1);
lcd.print(“Media=”);
x = y/nrox; lcdPrintFloat (x,1);

// ==== blinks led =============
digitalWrite(13, LOW); delay (100);
digitalWrite(13, HIGH); delay (100);

// ==== resetea sumas cuando nobs > 25 ==============

if(nrox==25){
y=0 ;
nrox=0;
}
}

/* -------------- Function LcdPrintFloat ---------------- */
void lcdPrintFloat( float x, byte precision){
// prints val on a ver 0012 text lcd with number of decimal places determine by precision
// precision is a number from 0 to 6 indicating the desired decimal places

if(x < 0.0){
lcd.print(’-’);
x = -x;
}

lcd.print ((long)x); //prints the integer part
if( precision > 0) {
lcd.print("."); //prints decimal point
unsigned long frac;
unsigned long mult = 1;
byte padding = precision -1;
while(precision–)
mult *=10; //“amplifies” decimal values

if(x >= 0)
frac = (x - int(x)) * mult;
else
frac = (int(x)- x) * mult;
unsigned long frac1 = frac;
while( frac1 /= 10 )
padding–;
while( padding–)
lcd.print(“0”);
lcd.print(frac,DEC) ;
}
}

Does anyone know how to convert it in a library or a kind of function one can call everytime it is needed?

AlphaBeta,

Thank you much.

I will test the functions you posted.

In the meantime, I made and posted a small sketch.

Your solution looks better and more compact indeed.

OldBeaver

AlphaBeta,

Couldn´t make your library to work.

I downloaded it and installed it. Called it in my routine, but my knowledge is too limited to make it work.

I made a small sketch that calculates the mean of every 10 data read.

But it is slow and not very practical. Would like to use your routine.

Would you please help me?

I published my routine in this thread.

OldBeaver

Try this:

// ============ Test Rutina para calcular media de n obs ========
//
// This routine computes mean of 10 obs of variable of interest.
// Includes function lcd
#include <LiquidCrystal.h>
#include <AverageList.h>

typedef float sample; //change float to the datatype you want to use
const byte MAX_NUMBER_OF_READINGS = 10;
sample storage[MAX_NUMBER_OF_READINGS] = {0};

AverageList list = AverageList( storage, MAX_NUMBER_OF_READINGS );

//create object to control an LCD.
LiquidCrystal lcd(12, 11, 6, 7, 8, 9, 10);

void setup() {
Serial.begin(9600);
}

void loop() {
// ============ generates random numbers for testing ======
sample rand = random(1,100)/1.5;
list.addValue( rand );

list.debug(“list”,Serial); //print contents to serial console
//list.debug(“list”,lcd); //print contents to the lcd

Serial.print("average: ");
sizeof(sample)==1 ? Serial.println(list.getAverage(),DEC) : Serial.println(list.getAverage()); //print bytes and chars as ints

delay(500);
}

And btw: use the [#] button when posting code :slight_smile:

The sketch I posted will print something like this:

list [51.33, 23.33, 52.00, 56.00, 57.33, 54.00, 16.67, 30.00, 36.00, 0.00]
average: 41.85
list [51.33, 23.33, 52.00, 56.00, 57.33, 54.00, 16.67, 30.00, 36.00, 25.33]
average: 40.20
list [48.67, 23.33, 52.00, 56.00, 57.33, 54.00, 16.67, 30.00, 36.00, 25.33]
average: 39.93

[edit]Download the latest library!
http://www.arduino.cc/playground/Code/AverageList[/edit]

AlphaBeta,

Thank you much for your sketch. I was waiting for testing it before acknowledge reception, but I will do the test later, probably tomorrow Monday. Today is a kind of relaxed time at home.

In the meantime, I read the sketch and found some sentences I haven´t seen before. Want to be sure I understand them.

For instance, you wrote:

AverageList list = AverageList( storage, MAX_NUMBER_OF_READINGS );

What is the meaning of “” ?
And of the complete sentence?

My intuition tells me that you mean “AverageList” is a subset of “list”, which is stored in storage ? Wrong?

Another sentence I don´t understand:

“sample storage” […] =

Haven´t seen this kind of command that have two words.

My intuition tells me that you are taking a sample from a set called storage and stored in [ … ] . Am I right?

Another sample of my ignorance: you wrote:

sizeof(sample)==1 ? Serial.println(list.getAverage(),DEC) : Serial.println(list.getAverage());

First of all “sizeof(sample)==1 ?” please explain this. Is it “?” part of the command or a typo mistake?

Furthermore, in the command:

Serial.println(list.getAverage()) I interpret the following:

You command the board to send and print results in a serial device, with the following parameters: “(list.getAverage())” where I suppose list is a set (you generated above in a line I didn´t understand, also), and getAverage() is a command of your new library that extracts the values of list.

I hope I can familiarise with the library, so I can sample and promediate data as it is captured.

My idea is:

Get a stream of data —> sample and promediate —> send info to next processing stage.

I need to do this for:

3 simultaneous analog sensors —>
3 simultaneous analog sensors —>
3 simultaneous analog sensors —>

4 simultaneous digital sensors —>
4 simultaneous digital sensors —>
4 simultaneous digital sensors —>
4 simultaneous digital sensors —>

And make some further calculations with the 7 resulting means, some calculation using several of them, mixing analog and digital in some cases. What I mean is that I must be very time-efficient with my sketch.

Please don´t send me the sketch for the 7 simultaneous sampling, as I wouldn´t understand it. Instead, I would thank you if you help me familiarise with one variable sampling, to find a way of programming the 7 sampling in the fastest way possible. Hopefully, make a funtion in my program that call some of your sample routine commands.

I think some of my results will be displayed in a LCD screen every one second (or even more frequently), and other every one minute.

Tks a lot for the extra help, in advance,

OldBeaver

Hello!

I see that I posted some unexplained code :frowning:
I’ll try to explain:

The first thing we need to clear out of the way is the use of the name sample.
This is just an alias selected by me, bacuase the AverageList library need to operate on the same kind of variable, so instead of writing int or byte (or something else) I created a type definition with the keyword typedef.
This makes for a more maintainable code. If, for instance, you where to change the datatype from say bytes to ints, you would only need to change the typedef line, as opposed to changing every instance of byte or int in your code.

I absolutely see now that I should’ve explained this, but I forgot it.


In the meantime, I read the sketch and found some sentences I haven´t seen before. Want to be sure I understand them.
For instance, you wrote:

AverageList list = AverageList( storage, MAX_NUMBER_OF_READINGS );

What is the meaning of “” ?

This wierd thing tells the compiler what kind of data you are going to be averaging.
If you see the line: typedef float sample; //change float to the datatype you want to use
You might guess that saple is exactly the same as float.


And of the complete sentence?

My intuition tells me that you mean “AverageList” is a subset of “list”, which is stored in storage ? Wrong?

Imagine this simple code:

int var = 5;

Here you say: create an integer called var and set it to 5.

The same sentance applies to the much more strangelooking:

AverageList<sample> list = AverageList<sample>( storage, MAX_NUMBER_OF_READINGS );

Here you can say: create an AverageList called list and set it to an AverageList that uses the storage array as storage, and has the size equal to MAX_NUMBER_OF_READINGS, which is 10.

The storage is just an array that allocates some memory that AverageList will use to store data.


Another sample of my ignorance: you wrote:

sizeof(sample)==1 ? Serial.println(list.getAverage(),DEC) : Serial.println(list.getAverage());

First of all “sizeof(sample)==1 ?” please explain this. Is it “?” part of the command or a typo mistake?

The ? is called the ternary operator, and is just a way of writing an if-else using as little typing time as possible.

condition ? code1 : code2;

Translates to: is condition true ? if yes execute code1 else execute code2

This: sizeof(sample)==1 ? Serial.println(list.getAverage(),DEC) : Serial.println(list.getAverage());
is exactly the same as:

if (sizeof(sample)==1){
  Serial.println(list.getAverage(),DEC);
} else {
  Serial.println(list.getAverage());
}

Furthermore, in the command:

Serial.println(list.getAverage())

First you tell the AverageList called list to calculate the average of the values currently logged, then you tell the Serial object to print them to the serial bus.


Please don´t send me the sketch for the 7 simultaneous sampling, as I wouldn´t understand it. Instead, I would thank you if you help me familiarise with one variable sampling, to find a way of programming the 7 sampling in the fastest way possible. Hopefully, make a funtion in my program that call some of your sample routine commands.

I changed the example to something simpler, let me know if something is unclear :slight_smile: :

#include <AverageList.h>

const byte MAX_NUMBER_OF_READINGS = 10;
float storage[MAX_NUMBER_OF_READINGS] = {0};

AverageList list = AverageList( storage, MAX_NUMBER_OF_READINGS );

void setup() {
Serial.begin(9600);
}

void loop() {
// ============ generates random numbers for testing ======
float rand = random(1,100)/1.5;
list.addValue( rand );

list.debug(“list”,Serial); //print contents to serial console
Serial.print("average: ");
Serial.println(list.getAverage());

delay(500);
}

Ok, AlphaBeta,

Little by little I am understanding it. I hope!

Then sample is a vector, an array of data, isn´t it?
Could be an array of integer numbers, float, or byte, as well.

By the way, I haven´t worked with byte numbers, but they seem to be like integers…

But, I see it is a clever way to name the type of numbers once from the beginning. Good.

The DEC word means that the data will be printed with decimals?

Well, I tried to compile the sketch and got several error messages:

  • Hardware\libraries\AverageList/AverageList.h:121: error: ‘byte’ does not name a type

  • Hardware\libraries\AverageList/AverageList.h: - in constructor ‘AverageList:: AverageList(datatype*,int)’

  • same …/ class ?AverageList does not have any field named ‘size’.

  • same…/ ‘currentIndex’ was not declared in this scope.

and several following errors saying “was not declared in this scope”

Some other error notifying some sintax errors too:

  • same (hardware) …/ AverageList.h:68 error: expected ; before ‘i’

I tried to find out the possible mistake, but couldnt find it.

One possible but little probable cause: I have never used the PC monitor to see results of the calculations. I have always used the LCD screen. Is it related to that, maybe?

Anyway, it is not a bad idea to learn using the notebook to watch results.

Looking forward to hear your advice,

OldBeaver

AphaBeta,

Your routine is working fine, sorry. I didn´t realise that you updated the AverageList library.

Once I loaded it, the sketch begun to work ok.

Thank you very much.

OldBeaver

Alpha beta,

Your last sketch is very concise.

Can you add a line where I can see the sampled values of which the routine calculates the average?

I would eliminate that line in the future, it is only for checking the values.

Thank you,

OldBeaver

Can you add a line where I can see the sampled values of which the routine calculates the average?

What exactly do you mean? :)

The sample value is called rand

sample rand = random(1,100)/1.5;

To print it, add:

sample rand = random(1,100)/1.5;
Serial.print("Random/sampled value: ");
Serial.println(rand);

AlphaBeta,

Well, sorry, I should explain better what I wanted.

The routine I am testing is:

// ======= Test Routine para calcular media de n obs ========
// --------------------- por AlphaBeta ----------------------
// Computes mean of 10 obs of variable of interest.
// Includes function lcd
#include <LiquidCrystal.h>
#include <AverageList.h>

const int MAX_NUMBER_OF_READINGS = 10;
float storage[MAX_NUMBER_OF_READINGS] = {0};

/* ------- create object to control an LCD GMD1602K -------*/
LiquidCrystal lcd(12, 11, 6, 7, 8, 9, 10);

AverageList list = AverageList( storage, MAX_NUMBER_OF_READINGS );

void setup() {
Serial.begin(9600);
}

void loop() {
// ============ generates random numbers for testing ======
float rand = random(1,100); list.addValue( rand );
lcd.setCursor(0,0); lcd.print(“Random:”); lcd.print(rand);
delay (5);

//lcd.setCursor(0,1); list.debug(“list”,lcd); delay (500);
lcd.setCursor(0,1); lcd.print("Average: "); lcd.print(list.getAverage());
delay(500);
}

So, converting your line to printing on the LCD screen, the line I have

lcd.setCursor(0,0); lcd.print("Random:"); lcd.print(rand);

should the sampled values. It prints something, but I am not clear what it is…

There is another line in the sketch, a debug printing line:

//lcd.setCursor(0,1); list.debug("list",lcd); delay (500);

When activated, this line prints a set of values within square parenthesis [ value, value, …]

As the LCD screen is small, only a few are shown.

However, some doubts:

a) Why the debug values printed seems to be always the same set?
b) The mean calculated is a moving average, (only one value changes each time the mean is calculated)? Or, it is a common mean (sum of 10 values divided by 10), where all the 10 numbers set is new?

Anyway, moving average would be very good too, maybe better, as a smoothing function.

Sorry for my limited English. Hope I explained well the issues.

Thank you for your valuable help,

OldBeaver

Dear AlphaBeta,

The program that you kindly sent to me is working. This is the sketch:

#include <AverageList.h>

const byte MAX_NUMBER_OF_READINGS = 10;
float storage[MAX_NUMBER_OF_READINGS] = {0};

AverageList<float> list = AverageList<float>( storage, MAX_NUMBER_OF_READINGS );

void setup() {
Serial.begin(9600);
}

void loop() {  
// ============ generates random numbers for testing ======
float rand = random(1,100);
list.addValue( rand );

list.debug("list",Serial); //print contents to serial console
Serial.print("average: "); delay(1000);
Serial.println(list.getAverage());

delay(1000);
}

I finally discovered how to print data to the PC monitor, so now I can see what the routine is doing.

It seems that it is populating the 10 memeber vector (list) and it calculates average everytime it add a new value.

What I don´t know is if it calculates the mean considereing the zeroes or not. I think it do not consider the zeroes. This mean that the first mean calculated is with one value. The second is with two values, until it populates completely the list with 10 values.

After that, every mean seems to be calculated with 10 values.

Besides, I discovered that the list changes one value at every time, so, the mean calculated is a moving average. That is Ok with me. Good!

From there on, I couldn´t realise what the sketch does.
At some point, it begins to produce some errors, which are displayed in red.

Summarizing:

Every loop the routine prints a line with 10 values, for which it prints the moving average.

But, something strange: the lines of previous sets of 10 values changed their values on the screen, like a matrix was formed. I expected that the results of the loop before the last loop should remain unchanged. But, it doesn´t at certain time.

One question:

What is the meaning of this line?:

float storage[MAX_NUMBER_OF_READINGS] = {0};

Ok, thank you much for your help. Looking forward to hear from you again soon,

OldBeaver

I finally discovered how to print data to the PC monitor, so now I can see what the routine is doing.

It seems that it is populating the 10 memeber vector (list) and it calculates average everytime it add a new value.

That is correct :)

What I don´t know is if it calculates the mean considereing the zeroes or not. I think it do not consider the zeroes. This mean that the first mean calculated is with one value. The second is with two values, until it populates completely the list with 10 values.

After that, every mean seems to be calculated with 10 values.

Spot on.

If you want it to cincider the zeroes, you can call getTotalAverage()

From there on, I couldn´t realise what the sketch does. At some point, it begins to produce some errors, which are displayed in red. ... But, something strange: the lines of previous sets of 10 values changed their values on the screen, like a matrix was formed. I expected that the results of the loop before the last loop should remain unchanged. But, it doesn´t at certain time.

Could you post a copy paste or an image of this? Very strange.

What is the meaning of this line?:

float storage[MAX_NUMBER_OF_READINGS] = {0};

This line initializes the storage to be all zeroes.

AlphaBeta,

Forget about the red errors: that was because I unplugged the board, so the system claims for a broken link on the serial port.

Will try to copy and paste results, and send them to you.

Thank you for the explanations. Every day one learn something new.

Old Beaver

AlphaBeta,

I cannot copy the results from the screen.

What really happens is as follows:

The list is displayed, each loop, followed by the moving average not considering the zeroes.

At the first loop all the values but the first are zeroes. The mean is 77.

In second loop all the values are zeros, but the first (77) and the second (35). The mean is now 56.

Third loop, the list is displayed, all the values are zeros, but the first (77), second (35) and third (78). The displayed mean is 63.33 .

And so on, until it fills the list completely with values. The 10th value is 38 and the calculated mean is 60.3 .

After that, the list remain unchanged (at least in the screen), but the mean continues to change: 59.9, 58.2, etc.

That is it.

I will test with getTotalAverage and see what happens...

Important:

Would you please send me the original sketch, as I changed it many times in many ways and now I am not sure that I am running the correct one.

Thank you very much,

OldBeaver


Dear AlphaBeta, Mem and the Forum,

Ok, tks to all of your explanations.

I will begin a new set of tests with one sensor at a time, today, to see if I can debug the multiple problems I am having with the car sensors and its processing.

This time, I will take advantage of the new knowledge I got from you people (AlphaBeta and Mem), from some books I got and from my own experimenting (trial and error, basically).

Let´s see if I succeed now. At least in part.

OldBeaver