Hi , this is the screenshot of the graph from my Arduino iot cloud dashboard. The data presented on the graph is water level in the lake, measured with ultrasonic sensor. Data is already filtered (Im using pingMedian from the NewPing library) and everything works great, Im getting smooth lines on the graph.
When there is a garbage under the sensor, plastic bottles floating etc. I normally get the spikes on the graph with wrong values as you can see in the attachment.
Is there some kind of filter I could use to discard this type of errors?
p.s. orange line on the graph is the real value.
a median filter followed by a low-pass would be approapriate
how many samples is your median filter?
the following is a common low-pass filter. it reaches close to the input value after 3N samples. the lake water level should be relatively constant to a large value of N would be appropriate (don't know your sampling rate which can also be low)
avg += (samp - avg) / N
2 Likes
A commonly seen expression makes it easier to see what the low pass filter, or "leaky integrator" is doing:
average = 0.9 * average + 0.1 * sample;
"The new average is mostly the old average, with a bit of influence from the latest sample."
0.9 and 0.1 add up to 1. In this case, @gcjr's N would be 10, and equivalent to using a weight of (N - 1) / N for the current average and a weight of 1/N on the latest sample.
For N = 5, we get 4/5ths and 1/5th, viz:
average = 0.8 * average + 0.2 * sample;
which is obvsly letting the latest sample have greater influence.
@gcjr's formula is mathematically equivalent to the more verbose expression, is more efficient and makes it easier to tinker with N.
a7
1 Like
common signal processing structure
2 Likes
Isolate the sensing zone within a cage to keep floating things out. This should also dampen the ripples.
1 Like
Yes I was thinking about that, but the place under the sensor is hard to reach...
if (millis() - startTime >= TIMEOUTDELAY)
{
unsigned int echoTime = sonar.ping_median(100);
delay(50);
// Send to Serial
Serial.print("<");
Serial.print(echoTime);
Serial.print(">");
Serial.read();
startTime = millis();
}
This is the code for the sensor section, every 6 seconds, I measure 100 times, get median and send the ping time to the UART.
I calculate the distance, based on ping time and send it to the cloud "on change".
BTW if you noticed there is a 1 hour period on the graph.
Out of context that code looks odd.
You have a common millis()-based timing mechanism, but what it controls has a call to delay().
The hole point of the mechanism is so you can avoid delays.
a7
And this is what I got when garbage went away.. it all works fine with the occasional value oscillating +/-1cm.
Yes you are right, delay should be there, but that is not the problem. What is out of context, this code is all I have on the sensor measuring module?
For all we know, what you did makes perfect sense in the context of the entire sketch.
Just seeing that function means we can only comment on what's in front of us, like your unusual timing timing.
a7
Sorry I dont understand. I need help, or idea, for postprocessing of the data I get from that function. Have you read the first post, or I wasnt precise enough? It`s about errors due to the garbage floating on the water.
You can't get away from LSB jitter in ANY digital signal.
RMS is the traditional signal amplitude computation.
Below sensor cage isn't needed or wanted. It might give false trigger. Just put cage around sensor and leave bottom open.
Yes, but LSB jitter is not my problem. The cage is out of the question, I need software solution.
I tried low pass filter long ago, but its not working good, the error acuumulates.. and what should be the frequency of the low pass filter? What about cascaded median? p.s. I could tolerate even few minutes of lagging of the result , dont need to know the value in real time..
can you post some raw samples?
1 Like
Maybe you will need to tolerate much more than a few minutes of lagging.
To get a handle on that, analyze your current data to see how long it usually takes a raft of garbage to pass by.
Using a median filter is probably an ok approach, but the total sample time needs to be long enough to exclude the garbage from the area around the median.
The NewPing library has a 29 ms delay between median samples...
...so if you do 100 samples, that's a total sampling time of only three seconds. That, and your six second TIMEOUTDELAY, are obviously not long enough.
After you analyze your current data and determine the typical time it takes for garbage to pass by, then you can adjust your TIMEOUTDELAY and the PING_MEDIAN_DELAY to suit.
Even so, some outliers will probably slip through. In that case, something like
average = 0.9 * average + 0.1 * sample;
should help.
You could "bench test" your new approach using Excel or similar with your current data.
Ok, thanks a lot for a few guide steps, thats what I need :) Ive started to learn arduino framework and C++ only year ago so, Im still new to terminology too.. Ill download csv file with data from last 24 hours and see what I`ll figure out..
The below code, for your reference, uses the ExponentialFilter to smooth a "noisy".
void ExponentialFilter(EXPFLTR *expFilter) {
static bool firstScan = false;
if (!firstScan) {
expFilter->output = expFilter->input;
firstScan = true;
}
if (expFilter->reset) expFilter->output = expFilter->initial;
if (expFilter->factor < 0.0f || expFilter->factor > 1.0f) {
expFilter->badPara = true;
return;
} else {
expFilter->badPara = false;
}
expFilter->output += expFilter->factor * (expFilter->input - expFilter->output);
}
#define resetPin A0
#define factorPin A1
#define noisePin A2
#define inputPin A3
#define interval 10UL // 10 milliseconds
typedef struct {
float input;
float initial;
float factor;
float output;
bool badPara;
bool reset;
} EXPFLTR;
EXPFLTR smooth;
float factor;
float noise;
long noiseScaleRandom;
float input;
float processValue;
void setup() {
Serial.begin(115200);
pinMode(resetPin, INPUT_PULLUP);
}
void loop() {
bool reset = !digitalRead(resetPin);
long factorAin = analogRead(factorPin);
long noiseAin = analogRead(noisePin);
long inputAin = analogRead(inputPin);
bool trigger = Interval(interval);
// Factor Simulator
long factorScaling = map(factorAin, 0, 1023, 1, 2000);
factor = factorScaling / 10000.0f; // Noise:= 0.0001 - 0.2000
// Noise Simulator
long noiseScaling = map(noiseAin, 0, 1023, 1, 100000);
noiseScaleRandom = random(0, noiseScaling);
noise = noiseScaleRandom / 10000.0f; // Noise:= 0.0000 - 10.0000
// Process Value Simulator
long inputScaling = map(inputAin, 0, 1023, 0, 10000);
input = inputScaling / 100.0f;
processValue = input + noise;
// Set the value to EXPFLTR parameters.
smooth.initial = processValue;
smooth.input = processValue;
smooth.factor = factor;
smooth.reset = reset;
ExponentialFilter(&smooth);
Plot(&smooth);
}
bool Interval(unsigned long intervalTime) {
static unsigned long start = millis();
unsigned long elapsed = millis() - start;
if (elapsed < intervalTime) return false;
start = millis();
return true;
}
void ExponentialFilter(EXPFLTR *expFilter) {
static bool firstScan = false;
if (!firstScan) {
expFilter->output = expFilter->input;
firstScan = true;
}
if (expFilter->reset) expFilter->output = expFilter->initial;
if (expFilter->factor < 0.0f || expFilter->factor > 1.0f) {
expFilter->badPara = true;
return;
} else {
expFilter->badPara = false;
}
expFilter->output += expFilter->factor * (expFilter->input - expFilter->output);
}
void Plot(EXPFLTR *expFilter) {
Serial.println("expInput:" + String(expFilter->input, 4)
+ ", expOutput:" + String(expFilter->output, 4)
//+ ", expFactor:" + String(expFilter->factor, 4)
);
}
MicroBeaut (μB)
1 Like
Just a quick question, Im using ping_median(100); ,so what about 500 samples, is the memory of the chip a main limitation for number of samples? btw. Im using ESP8266_01 for this.
raw_data_uss.zip (55.1 KB)
This is the excel file of raw data, I`ve added chart for a time period of interest.