Help with signal conditioning

I wonder if somebody can point me to a software algorithm or an hardware solution to obtain nice square waves (sharp attack, sustain and, what's missing here, a sharp release).

Fundamentally, there is no good solution.

Alternatively, you can introduce some derivative gain, similar to a PID controller, for this.

Rob,

Does it solve your problem now?

I have to try in practice in my application see if the result is acceptable.
In a "perfect" wind controller, I should be able to detect a gradual decrescendo, that is the player blowing softer and softer to make the sound fade away gradually but I don't know how I would differentiate that from the slow decay that I see when I stop blowing altogether.

Otherwise you might need the second derivative too.

By second derivative you mean something like:

newderivative = oldvalue - newvalue ;
secondderivative = oldderivative - newderivative ;
oldderivative = newderivative ; // Remembers the previous one

Missing in the graph is the square wave, derived from the first derivative. Would make the picture complete.

Sorry, I did not include it because it does not show very well, But I am attaching it now. I am also attaching the actual Excel in case you are interested (the 8 periods moving average was calculated in the spreadsheeti itself).

Maybe smoothing the original signal with a moving average is better than smoothing the derivative (should become smoother automatically).

I did not think about that because the actual signal seems clean enough but maybe a 2 or 4 period moving average of the signal will yield a clean derivative as well and be faster to show a true change

seen this class - Arduino Playground - RunningAverage - ?

No, I haven't. It does look very useful. :slight_smile:

My approach is very similar to yours in:

long runningAverage(int M)

which avoids the need for floating math.

with the difference that I use two distinct routines, one for initializing the array during setup() and one to update the actual running average from loop().
That has the advantage of eliminating

  if (count < LMSIZE) count++;

so that the division is always by 16 (in my case) which could be replaced by a rightward 4 bit shift (though if I understand correctly, the compiler is smart enough to make the replacement).
Also, is:

  index = index % LMSIZE;

More efficient than:

 index++ ;
if (index >= LMSIZE) index = 0 ; // If LMSIZE is 16 index cannot be more than 15

I can post the code for my little functions if of interest (and if you'd like to comment on it)

Thanks again

Moving Average.xls (105 KB)

newderivative = oldvalue - newvalue ;
secondderivative = oldderivative - newderivative ;
oldderivative = newderivative ; // Remembers the previous one

Yes that should work.

Also, is:

 index = index % LMSIZE;

More efficient than:

index++ ;

if (index >= LMSIZE) index = 0 ; // If LMSIZE is 16 index cannot be more than 15

No, shorter source code does not always mean shorter exe or faster runtime.
The modulo is a division which is one of the most expensive basic math operations.

Yes, please post the code, you want to share, I'll comment (maybe others too :wink:

In the image I see quite some HIGH LOW fluctuations, e.g between [2.2 , 3.6] smoothing the source signal should really help here.

You can make the square wave higher of course by writing 400 iso 100

  // SQUARE OUTPUT (% as example)
  if (derivative > 0) Serial.println(400);
  else Serial.println(0);

dhenry:
Alternatively, you can introduce some derivative gain, similar to a PID controller, for this.

I find your suggestion interesting.
I found that:

While the math is way above my head, in the bottom they put a simple pseudocode, which they said is suited for MCUs:

previous_error = 0
integral = 0 
start:
  error = setpoint - measured_value
  integral = integral + error*dt
  derivative = (error - previous_error)/dt
  output = Kp*error + Ki*integral + Kd*derivative
  previous_error = error
  wait(dt)
  goto start

but I am not sure about the whole idea behind it though it resembles the derivative idea of Rob.

Also, how to determine the dt (time between measurements) given that I need the fastest response possible. Should I eliminate the "wait" statement and make dt just = millis() - last_millis() ?
Kp, Ki and Kd are tuning constants and that, I suspect, is more a matter of trial and errors.

How could I apply it to my problem (slow decay curve)?

Thanks

check - Arduino Playground - PIDLibrary -

robtillaart:
check - Arduino Playground - PIDLibrary -

Boy, I definitely have to spend more time in the playground :).

Though I still don't know how PID helps me.
In other words, I don't know what the note should be, the input value is supposed to tell me that.

Am I missing something?

Thanks

robtillaart:
Yes, please post the code, you want to share, I'll comment (maybe others too :wink:

Here it is (I wanted to triple check it and test it before posting):

/*  Breath Sensor and Moving average
    By Luca Brigatti
    Adapted from an algorithm by robtillaart
    Calculates the derivative (difference) between two subsequent sensor readings
    and calculates a moving average of the derivative signal
    Optimized for a running average of 2^n elements ( 2, 4, 8 , 16 etc.)
    Sensor used for testing is breath sensor by Modern Devices: 
    http://shop.moderndevice.com/products/wind-sensor
    
*/

const boolean DEBUG = true ;  // true, to print data on the serial port. false to do something else
const int MAVSIZE = 16 ; // Elements (size) of the moving average

// Signal variables
int oldval ;
int newval ;
int derivative ;  // will hold oldval - newval

// Moving Average variables
int MavValues [MAVSIZE] ;  // Holds the last MAVSIZE values of the moving average
int mavtot = 0 ; // total of the moving average (before the division)
int mav = 0 ; // Moving average = mavtot / MAVSIZE
int mavindex = 0 ; // next value to use for the moving average
int sizemask ; // Mask to use for the index counter


void setup()  {

  Serial.begin(9600);  // <--- Low speed: easier to collect data from the serial port, High Speed: smoother curve and tighter moving average
  sizemask = MAVSIZE -1 ;  // If MAVSIZE is B10000 (16), sizemask is B01111 (15)
  setupmav  (MAVSIZE) ; // Initialize the Moving Average with the first MAVSIZE values
  if (DEBUG) Serial.println("Square breathing 0.1");

}

void loop()  {

  newval = analogRead(A2);  // measure the wind
  derivative = (newval - oldval);  // first derivative of the signal
  updatemav (MAVSIZE) ;  // Updates the moving average, a new mav value is assigned.
  
  if (DEBUG) serialdump() ;
  else ; // Do something with mav , the moving average of the derivative or newval, the actual reading
  
  oldval = newval ; // Remember previous value for next derivative

}

void serialdump () {

  // OUTPUT CAN BE COPIED TO EXCEL
  // Put here what you want to print
  Serial.print(millis());
  Serial.print(";");
  Serial.print(oldval);
  Serial.print(";");
  Serial.print(newval);
  Serial.print(";");
  Serial.print(derivative);
  Serial.print(";");
  Serial.print(mav);
  Serial.print(";");

  // SQUARE OUTPUT (% as example)
  if (derivative > 0 && newval > 40) Serial.println(100);
  else Serial.println(0);

}

void setupmav(int mavsize) {

  mav = 0 ;
  mavtot = 0 ;
  oldval = analogRead(A2) ;  // Starting point, change with Pin actually used
  
  for (int i = 0 ; i < mavsize; i++ ) {

    newval = analogRead(A2);  // measure the wind, change with pin used
    derivative = (newval - oldval);  // first derivative of the signal
    MavValues [i] = derivative ;  // Remember this value
    mavtot +=  derivative;  // Update the running total
    oldval = newval ; // New Value is now old value

  }

 mav = mavtot / mavsize ; // First moving average

}

void updatemav (int mavsize) {

  mavtot += derivative ;  // Add latest value
  mavtot -= MavValues [mavindex] ;  // Subtract oldest value in the array
  MavValues [mavindex] = derivative ;  // Newest value in array, replaces oldest value
  mav = mavtot / mavsize ; // New moving average
  mavindex++ ; // Point to the next element in the array, now the oldest element.
  mavindex &= sizemask ; // Circle back to 0 if reached size of the moving average  (e.g.  10000 & 01111 (16 & 15) = 00000 (0)  )

}

My algorithm is not as abstracted and nearly as useful as your library but it is optimized for a situation like the one at hand:
Putting the initial loading of the moving average array in setup and the main moving average calculation in the loop function eliminates the need for an "if" statement in the main loop to keep the count on how many values are in the array.
Also, this algorithm uses only integer math.
Finally

mavindex++ ; // Point to the next element in the array, now the oldest element.
  mavindex &= sizemask ; // Circle back to 0 if reached size of the moving average  (e.g.  10000 & 01111 (16 & 15) = 00000 (0)  )

Replaces your modulus and my

 if (mavindex=16) mavindex = 15 ;

As I feel it may be faster than both (but I really don't know).

Your comments are welcome.

A fast moving average would be something like this:

mav = (1-alpha)*mav + alpha * new_data;

where 0<alpha<1 as a weight for new observations. The smaller alpha is, the longer memory mav has of historical data.

If you pick alpha to be 1/2^n, the math gets considerably easier as you can simply shift. For example, for alpha = 1/16:

mav = mav + (new_data - mav) / 16.

essentially, 1 subtraction, 1 shift, 1 increment.

dhenry:
A fast moving average would be something like this:

mav = (1-alpha)*mav + alpha * new_data;

where 0<alpha<1 as a weight for new observations. The smaller alpha is, the longer memory mav has of historical data.

If you pick alpha to be 1/2^n, the math gets considerably easier as you can simply shift. For example, for alpha = 1/16:

mav = mav + (new_data - mav) / 16.

essentially, 1 subtraction, 1 shift, 1 increment.

That's very elegant.
It eliminates the need for an array to keep track of the last "n" numbers (and the ++ and &= operations on the counter) and eliminates the initializing function altogether.

Thanks!

P.S. Question for the C++ wiz. Would:

mav += ((new_data - mav) / 16) ;

work? faster? are the external () necessary?

P.S. Question for the C++ wiz. Would:

mav += ((new_data - mav) / 16) ;

work? faster?

You can easily test if this works (depending on the datatypes you can have certain side effects !)
faster? think yes you need to measure 10000 times or so to see diffs.

robtillaart:

P.S. Question for the C++ wiz. Would:

mav += ((new_data - mav) / 16) ;

work? faster?

You can easily test if this works (depending on the datatypes you can have certain side effects !)
faster? think yes you need to measure 10000 times or so to see diffs.

So would this in effect be acting as a low pass filter on the signal? If so how is the 16 related to the 'corner frequency of the effective filtering action, or is that more a function of how often the statement is executed per time unit?

Lefty

retrolefty:
So would this in effect be acting as a low pass filter on the signal? If so how is the 16 related to the 'corner frequency of the effective filtering action, or is that more a function of how often the statement is executed per time unit?

Ah... I yield to dhenry and robtillaart (not that I would not know the answer....) :blush:

mav += ((new_data - mav) / 16) ;

Better yet:

mav += ((signed short) (new_data - mav)) >> 4;

Before you get too excited about this approach, it has limitations:

It requires that the new_data to be greater than 1/16th. Otherwise, the moving average will degenerate to 0. One way to avoid this is to use fixed point math. For example, use the last two four (binary) digits as "decimal points":

  new_data=analogRead(MY_CHANNEL) << 4; //read the data
  mav += ((signed short) (new_data - mav)) >> 4; //compute moving average

You just need to recognize that mav is 16 times bigger.

You can also multiply the reading by 100 (for example) to obtain two decimal points.

The key is to avoid degeneration with small numbers.

@ dhenry.

Thanks for the clarification!

In my particular case, new_data is often = 0 as it is the difference between two sensors values taken at very short interval, so it won't work for me.

your original suggestion:

mav = mav + (new_data - mav) / 16 ;

is not affected by this problem, or is it?

Thanks

There are two issues: signed vs. unsigned, and integer math.

Which one are you referring to?

dhenry:
There are two issues: signed vs. unsigned, and integer math.

Which one are you referring to?

The value I am averaging is a signed integer which varies roughly between -5 and +5 (but in theory could be more) and the 16 periods moving average (in my tests) is very often 0 with some peaks between -2 and +2.

Thanks

Yes, it will have issues with long memory: the smoothed signal would accurate reproduce the peaks / valleys.

Two solutions:

  1. use shorter memory: rather than 1/16th, use 1/8th, or 1/4th. The data generated this way will be more volatile.
  2. scale up observations: rather than having data series in -5 - +5 range, scale them up by 10 (or 16, using left shifts) for example. The resulting moving average will be correspondingly scaled up.

Thanks!

Just wanted to report on my results, for whomever may be interested.

I applied a simple "decision" to the 16 period moving average:

IF moving average > 0 I am blowing on the sensor
IF moving average < 0 I am NOT blowing on the sensor

with the corollary that:

IF moving average == 0 I am doing whatever I was doing before // No need to actually state

This is the result in graphic form:

The blue line is the sensor data, the red line is the moving average (multiplied by 100 for ease of plotting) and the green line is the result of the decision: 100 I am blowing, 0 I am not blowing.

It's getting accurate, though there are some false positives and false negatives now and then and, in this very example, there are 86 ms of delay from the beginning of the decay (when I stop blowing) to the response of the moving average (when the software realized that I stopped blowing).

I'll continue tweaking the software.

If there is interest I'll keep posting updates.