On my Arduino Dash project, I'm looking to measure a few sensors on a very slow rolling average. Things like air temperature, oil temperature and fuel level.
For these sensors, I really want to avoid spurious readings, fast moving numbers and fuel sloshing around, so I want these numbers to creep up and down as they go.
This is out of the range of things libraries like analogsmooth as the times are much much larger. I guess I would want maybe 50-100 readings over maybe a minute and to roll the average along. What would be a good way of doing this?
If you want to avoid spikes and other noise, low-pass filter the readings, so any noise in the readings themselves is removed, and the response is slowed down to whatever response time you want.
This will give you a nice, smooth response, with no spikes, unless your raw readings are REALLY noisy. To increase filtering/slow response, make filterK smaller. To reduce filtering/speed response, make filterK large, up to a maximum of 1.0, which will do NO filtering at all.
Well with the fuel level particularly, the fuel its self will be sloshing around so the input data will be all over the place. The code needs to heavily damp the response so the needle stays rock solid and slowly moves its position in line with the actual tank level.
In the case of the temperature sensors it's less lumpy data, but to not be distracted by the coolant temperature needle wobbling or moving quickly I'd like it to also be really heavily damped so it moves no faster than 5 or so degrees per second at an absolute maximum.
Would a pass filter do this, or is it simply cutting out spurious values?
jremington:
Try an "exponential filter" with a long time constant, e.g. alpha = 0.01 or smaller.
Less than optimally coded, but readable example. All variables are floats.
For example, a simple moving average filter: SimpleMovingAverage.ino
It also comes with an efficient integer implementation of of the single-pole exponential filters proposed in the previous replies: EMA
You can find an explanation of the differences between a simple moving average filter (SMA) and an exponential moving average filter (EMA) here: Simple Moving Average Exponential Moving Average
These pages also contain a more in-depth explanation of the implementations used by the library.
If you get a lot of outliers that are not filtered out by the low-pass filters, you can combine it with a median filter as well: MedianFilter.ino
Wow lots of good information there, thanks Pieter. It's late here so I'll have a proper read in the morning, but given my application which method would you recommend given your extensive knowledge of this subject?
I'm hoping not to have many outliers so it's more of a damping effect I'm looking for.
It seems to do similar to what I'm looking for with slightly noisy data from my test board, but outliers do cause it to flicker a little when linked to a scaling bar on my arduino OLED.
It depends. The EMA is much more efficient, because it just requires a single state variable, regardless of the cut-off frequency. The SMA on the other hand requires N state variables. For low cut-off frequencies, N will be very large.
The EMA weights the samples exponentially, that is, recent samples have a greater effect on the filtered output than old samples. The SMA weights all samples equally (only the N most recent samples, of course).
Neither of those two filters performs especially well, the roll-off is slow, neither has a flat pass-band. The SMA has a lot of stop-band ripple, but does have a linear phase response. The EMA doesn't have any ripple, but being a first order infinite impulse response filter, it lacks a linear phase response.
Signal processing 101:
An ideal filter has very quick roll-off: all frequencies in the pass-band remain unaltered, and all frequencies in the stop-band are attenuated completely, and the transition between the pass-band and stop-band is immediate. However, these kinds of filters are impossible to implement in real-time, so all physically realizable filters will have a transition between the pass- and stop-bands. The steeper the roll-off, the closer it is to an ideal filter.
The following figure shows the frequency response of an ideal low-pass filter:
And these are the frequency responses of the SMA and EMA filters with a certain cut-off frequency (the red line):
As you can see, low frequencies pass through the SMA and EMA fine, and higher frequencies are attenuated, but the response is nowhere near the ideal filter response. The SMA has a pretty strange response in the stop band: some frequencies are attenuated completely (-∞ dB), while at other frequencies, a larger part of the signal is let through. This is called ripple.
The phase response shows how much certain frequencies are delayed. If the phase response is linear, all frequencies are delayed by the same amount. If this is not the case, you get phase distortion, i.e. the shape of your signal changes. You can see that the SMA has a linear phase response, but the EMA has not.
If you just want to smooth some sensor values, you probably don't care, but it is important when dealing with things like audio, ECG signals, etc. The steepness of the roll-off might be an important factor though. You get steeper roll-off by using higher order filters. In the case of the SMA, you can increase N (this requires more memory). If you increase the order of the EMA, it's no longer an EMA, EMA is always first order, but there are other infinite impulse response filters of higher order if you need them, such as Butterworth filters, Chebyshev filters and Elliptic filters.
For your application, I'd experiment with an EMA first, it's the goto approach for simple smoothing. An SMA is easier to understand and reason about (it's clear what kind of effect increasing the number of samples has) but unless you really need linear phase, it's not really a great filter, especially since it requires so much memory.
You can experiment with different filters to see which ones you like best.
A median filter is a completely different kind of filter, it's nonlinear. It simply keeps track of the N last measurements, sorts them, and returns the median value (the one in the middle). This eliminates outliers, but introduces around N/2 samples of delay. It's also much less efficient, because it requires N state variables, and sorting the samples at each time step takes time.
As ever, start simple and add complexity if you have to. I'd just try the leaky integrator already suggested initially and see how it does. How much fuel slosh will there be?
I forgot (if I ever knew) what kind of vehicle the dash will be fitted to. Can you assume that the reading at power up is likely to be good with minimal sloshing until you get moving? If so, it might be worth calculating from initial fuel and RPM over time, how much fuel you should have now and reject values that are way off. You would have to account for refueling of course.
Make sure that you hide the fuel level handling away in a function or class of its own so you can swap it out for a different algorithm easily.
AFAIK, the term "leaky integrator" refers to an analog low-pass filter.
The digital or discrete-time counterpart is called an exponential moving average (EMA) or exponentially weighted moving average (EWMA), often referred to as a "digital single-pole filter", because it's the simplest IIR filter with just one pole.
