What filter / algorithm to use ?

[gcjr]

@noobmastha

ping_median(100)

from NewPing

• sonar.ping_median(iterations [, max_cm_distance]) - Do multiple pings (default=5), discard out of range pings and return median in microseconds. [max_cm_distance] allows you to optionally set a new max distance.

sounds like this already filters out extreme values (i.e. median filters).

@noobmastha

In about 15 seconds I can get 5 values, then do median of these 5 samples.

not sure how a 5-pt median filter can be applied to sample set of 5 samples. it would result in one value, the median value of the 5 pts or 5 samples with the same value.

@noobmastha

at last, apply low pass filter .

the resulting one value if then averaged with the value from the preceding measurement cycle.

[gcjr]

here's a log10 histogram of the sample-to-sample variation.

there were ~10^3 (1000) samples that had < 0.01 variation and the same # of samples between 0.01 -0.02.
~3 (10^0.5) samples between 0.10 - 0.15
and trending upward > 0.15 variation

so maybe any measurement > 0.15 should be ignore or repeat the measurement until < 0.15

[DaveEvans]

@gcjr

sounds like this already filters out extreme values (i.e. median filters).

It takes samples every 29 ms (unless the default sample time is changed... which is certainly possible).

So, as mentioned in #16, 100 samples are taken in about 3 seconds.

Is that sufficient time for the trash (or whatever) to clear the sensor FOV and allow the median to represent the water surface elevation?

[DaveX]

Note that there might be a ton of missing data.

A cheap histogram of the first 200 time intervals lists gaps up to 800s:

On the whole set, gaps up to 3500sec:

(truncated above 20 observations)

You can see some of the gaps glossed over in this graph:

@noobmastha

And this is what I got when garbage went away.. it all works fine with the occasional value oscillating +/-1cm.

That time-smoothing interpolation covers lots of issues.

[noobmastha]

Is that sufficient time for the trash (or whatever) to clear the sensor FOV and allow the median to represent the water surface elevation?

No its not sufficient , thats why I`m doing this, but you gave me an idea. I can set more time between samples to get wider window.

[DaveEvans]

@noobmastha

I can set more time between samples to get wider window.

Yes, as suggested in #16...

[fobi123]

@noobmastha

This is the best solution, IMO, as you said, median followed by simple low-pass filter, I`ll implement that and get back with the results..

IMO the best solution is peak detector followed by LPF. Here is 3 points median, vs LPF vs peak.

Cheers

[noobmastha]

@gcjr

not sure how a 5-pt median filter can be applied to sample set of 5 samples. it would result in one value, the median value of the 5 pts or 5 samples with the same value.

Sorry, i just need some terminology clarification, what is "5-pt median", not a median of 5 samples?

[mikedb]

@westfw

Internet round-trip time measurement will discard "obviously wrong" data. If you're expecting data that slowly changes, with noise of +/- 0.1, and you get a sample that is off by 2 (or two standard deviations, or whatever), you just throw it away.

I think you should explore this think out of the box use some common sense no math is going to solve your problem.

The level will rise and fall slowly every sudden rise or drop must be discarded compare new reading with avg value if more than 5% throw away.

On power up do 100 readings in Setup get an Average , 100 readings done continue to void loop()

The Math is need up to a point then human intervention will filter the rest.

[gcjr]

@noobmastha

what is "5-pt median", not a median of 5 samples?

an N-pt median filter results in the median value of the last N samples. a 3-pt filter would ignore a single sample that is extreme. a 5-pt filter would ignore 2 consecutive extreme samples.

and filtering is generally applied to samples collected periodically.

@noobmastha

In about 15 seconds I can get 5 values, then do median of these 5 samples.

i hope you see that a median filter of these 5 samples is one value, their median value, let's call is a super-sample. of course the median value can also be taken of the last 5 super samples

(i'm guessing that a piece of garbage can affect all samples collected in a 15 second period)

but it seems you collect samples in bursts (see histogram) a number of samples are collected in a short period of time from which a single value is determine that is then filtered again.

but even the sampling bursts don't appear periodic.

it's not clear why you don't just collect a sample or even a super sample consistently once every minute, for example

[noobmastha]

@gcjr

it's not clear why you don't just collect a sample or even a super sample consistently once every minute, for example

That`s the plan as @DaveEvans suggested.

@gcjr

(i'm guessing that a piece of garbage can affect all samples collected in a 15 second period)

Yes, sometimes a few minutes can pass, before I get the right value.

[DaveX]

@noobmastha

Yes, sometimes a few minutes can pass, before I get the right value.

So, what filter/algorithm is used to produce this censored input data?

Please post your code.

[noobmastha]

post #7

[DaveX]

@noobmastha

if (millis() - startTime >= TIMEOUTDELAY) 
  {
    unsigned int echoTime = sonar.ping_median(100);
    delay(50);
    // Send to Serial
    Serial.print("<");
    Serial.print(echoTime);
    Serial.print(">");
    Serial.read();
    startTime = millis();
  }

That's the code? Is something fudging with TIMEOUTDELAY to make it non-constant in the 10-800sec range?

You have noisy data in both measurements and timing, If you carefully train your filter to get all the garbage out on this set, it might behave unexpectedly when the weather isn't calm.

[dlloyd]

Here's a plot of your raw data (green) from post #20 compared with a plot (orange) that just limits the change to ±0.02 and averages the previous 8 samples.

filter.ino - Wokwi Arduino and ESP32 Simulator

[noobmastha]

Wow, thank you a lot for your time, making the example in Wokwi.

[dlloyd]

Needs a true circular buffer that controls the head/tail indexes and reads correctly (or fix some other issue), then we can avoid this (and others). Might be some other issue, but too busy right now - might get a chance to take another look at this later...

[build_1971]

But the low pass has a phase shift...

[build_1971]

I think the median lpf is a good one. You might consider implementing both with unsigned integers instead of floats. Comparison (in your sorting algorithm) and division will be way faster (especially if you divide by some power of 2 (i.e. 8).
The leaky integration van also be implemented that way:

avg = (meas + 7* avg + 4) / 8;

The compliler will implement this as 3 bitshifts. The 4 is there to get the rounding right.
It also saves ram (not really problem with 5 pt. median, but considerable saving in n = 32 moving average).

[dlloyd]

Circular buffer now implemented in above Wokwi example ...