running Mean, Median or Mode - effective deviation detection

Hi -

I,m working on detecting foreign particle in a moving tubular forming at a speed of 10meters/sec, stuffed with material (within a range of dielectric constant but changing density in specific pattern - a picture is enclosed) by measuring the dielectric constant.

The specific change of stuffing density - requires to measure the running mean within a threshold band, and any foreign particle there is a small change in dielectric constant but changing density makes it challenging detection if a threshold with maximum and minimum is set.

Running average - doesn’t show up small changes in dielectric constant, wondering what would be most effective parameter to track this.

Running mean, median or more - what could be the best way to detect fast small change in dielectric constant

I’m using Arduino due, 12 bit resolution, analog input for the signal from the sensor.

Any suggestions appreciated.

requires to measure the running mean within a threshold band, and any foreign particle there is a small change in dielectric constant but changing density makes it challenging detection if a threshold with maximum and minimum is set

I don't think anybody can tell you what data/information you need or how it should be analyzed... I think you need to collect some data and look at it, maybe load it into a spreadsheet, and decide what's useful to you.

From what you're saying, it sounds like there are minimum & maximum limits, and maybe the average is important. If it's a continuous process, of course it has to be a moving average, and you'll have to determine the time-duration (or number of measurements in the moving average).

At 10 meters per second, how many readings per second do you need to take? Is the Arduino fast enough, and do you have enough memory to store the required data?

The median may not be useful... As you probably know, the median won't necessarily change with data changes... If your boss gets a salary increase from $100k to $100 million, the mean/average salary in your company will jump by a lot, but the median won't change at all (assuming the median is under $100k :wink: ). And, the mean for a continuous process may be "difficult" to track.

It sounds more like a Digital Signal Processing (DSP) problem. You want to filter out certain frequencies of variation and enhance certain other frequencies.

Once you have an idea of what frequencies you want to filter, then a filter design tool like can be used to write the code to copy into your Arduino program. Even though the filter is designed in Hz, you can design your frequency in meters if you like.

As mentioned I'm using Arduino Due, my sample requirement is around 5000sps, Due sampling as I've seen in some of the forum messages captioned below, seems to be comfortable, my sample duration is 2 msecs, I'm planning to gather 5 sample sets all in all 10 msecs of max and min values as reference, compare it with next 10ms max-min - a sort of running max, min evaluation with threshold around them. Hope this would help my purpose.

/by default, I get around 25000 samples per second, readings around 570 ( 10bit ) with a hall sensor plugged in

//adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX2, 1); //~313000 samples per second, A/D readings kaput
//adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX
2, 2); //~288 000 samples per second, A/D readings now have about 10% deviation
adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX2, 3); //~267000 samples per second, A/D readings OK, the best compromise I got
//adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX
2, 4); //~220000 samples per second, A/D readings OK
//adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX2, ADC_STARTUP_NORM ); //~65000 samples per second,A/D readings OK
//adc_init(ADC, SystemCoreClock, ADC_FREQ_MAX
2, ADC_STARTUP_FAST ); //~47000 samples per second, from the name, I was tempted to think this would perform better. But I know nothing.

OK, so you're sampling 5kHz. What frequency range do you want to exclude from your analysis? 0 to 200Hz? Go to the link I gave earlier and select one of the predefined filters, either high-pass or band-pass. Then change the values to suit your application.

The filter designed by that script will contain a number of "taps" which is the number of samples that it must store to make the filter work. This is similar to your idea of collecting 5 samples and then analyzing them. It's going to suggest a number like 19, depending on what you're asking it to do. But you don't have to wait for all 19; you can get a valid output after every sample.

If you were just looking to calculate the moving average, there's simpler filters. But you're looking to remove the average, which is like removing the low frequencies from the signal.

I,would be requiring atleast 10 filter bands - my signals are analog voltages (not frequency),and number of samples is 5000samples/second, any voltage outside the filter bands need to trigger an output

“Bands”? You mean you have 10 sensors looking for the contaminants? Then you need 10 filters.

Yes, you’re looking for an amplitude, measured in volts or whatever. But your requirement (as I understand it) is to remove the average level and detect short pulses. That means you must do some processing in the frequency domain because “short” isn’t a voltage.

only one sensor - but it gives different dielectric constant values depending on the ingredients(will have mix of ingredients with this product), hence I, need to create multiple reference(bands) to store them to separate variables and create a threshold for each variable, any other material creeping in will be outside these bands and gets detected

That seems like the opposite. You want to calculate an average of the most-recent group of samples and sound an alarm when the average moves away from the desired reference value. You want to ignore small variations, like a single particle of the wrong stuff that sends your sensor off to a different value but it immediately goes back. Am I right?

That would be a low-pass filter: it ignores short period variations but slow changes are detected precisely.

A moving-average filter is easy to understand, easy to implement and extremely efficient. It should be no problem running the filter at 5kHz.

There's a great online book on DSP which helped me a lot. The Scientist and Engineer's Guide to Digital Signal Processing By Steven W. Smith, Ph.D. You want to jump straight into Chapter 15 - Moving Average Filters

Then you want to jump to the last section of that chapter which tells you how to implement the filter very efficiently. [Keep an array of the last N values and a total. To add a value, subtract the oldest value from the total, add the new value. Save that new value in the array. If you are ever asked for the result, return Total/N]

How many threads are you going to start, on the same day, for what appears to be the same project?

There is this thread and:

Divide and conquer is not a good strategy to employ on a message board. People come to help, not waste their time.

Project could be same - but the information requested are totally different, evaluating different possibility definitely not to waste anybody time.

This sounds a lot like the TCP Round Trip Timing problem, widely written about in networking literature.
Basically, you keep a running mean value, as well as a running standard deviation value. Excursions of the expected value from the mean, beyond some multiple of the standard deviation, indicate that you have something happening (different fluid, in your case, change in congestion in the networking case.) Too big a multiplier, and you chalk it up to some sort of temporary error. Neither event nor error samples are permitted to modify the running average (although you might want separate means for the alternative states.)


then you get to play the usual tricks to try to keep the calculation of running average a std “efficient.” and such.