Filtering only big changes (Processing)

Hi, I am trying to filter some values read from my Arduino and sent to my Processing sketch, everything seems to work fine except that I am getting really ugly signals that just go up and down, but only in small differences, so what I wanted to do is to tell the sketch "only change the value if it is over a +/- 50 threshold", I did this code but It doesnt seem to work:

import processing.serial.*;
 
 Serial myPort;        // The serial port
 int xPos = 1;         // horizontal position of the graph
 
 void setup () {
 // set the window size:
 size(800, 600);        
 
 // List all the available serial ports
 
 myPort = new Serial(this, "COM4", 115200);
 // don't generate a serialEvent() unless you get a newline character:
 myPort.bufferUntil('\n');
 // set inital background:
 background(0);
 }
 void draw () {
 // everything happens in the serialEvent()
 }
 
 void serialEvent (Serial myPort) {
 // get the ASCII string:
 String inString = myPort.readStringUntil('\n');
 
 if (inString != null && inString.length() > 0) {
 // trim off any whitespace:
 inString = trim(inString);
 String [] inputStringArr = split(inString, ",");
 
 // convert to an int and map to the screen height:
 float [] inByte = new float[5];
 inByte[0] = float(inputStringArr[0])-600; 
 inByte[1] = float(inputStringArr[1])-600;
 inByte[2] = float(inputStringArr[2])-600;
 inByte[3] = float(inputStringArr[3])-600;
 inByte[4] = float(inputStringArr[4])-600;
 
 inByte[0] = map(inByte[0], 0, 1023, 0, height);
 inByte[1] = map(inByte[1], 0, 1023, 0, height);
 inByte[2] = map(inByte[2], 0, 1023, 0, height);
 inByte[3] = map(inByte[3], 0, 1023, 0, height);
 inByte[4] = map(inByte[4], 0, 1023, 0, height);
 
 float [] inByteOld = new float[5];
 
for (int i=0; i<5; i=i+1)
{
  inByteOld[i] = inByte[i];
  if ((inByteOld[i] - inByte[i]) > 50 || (inByteOld[i] - inByte[i]) < -50)
  {
    inByteOld[i]=inByte[i];
  }
  else
  {
    inByte[i]=inByteOld[i];
  }
}
  
 
 
 // Digital filter of noise
 
 
 
 // draw the line:
 stroke(127,34,255);
 line(xPos, height, xPos, height - inByte[0]);
 stroke(255,0,0);
 line(xPos, height, xPos, height - inByte[1]);
 stroke(0,255,0);

 // at the edge of the screen, go back to the beginning:
 if (xPos >= width) {
 xPos = 0;
 background(0); 
 } 
 else {
 // increment the horizontal position:
 xPos++;
 }
 }
 }

thats the whole code, the part I think should filter this treshold is this:

for (int i=0; i<5; i=i+1)
{
  inByteOld[i] = inByte[i];
  if ((inByteOld[i] - inByte[i]) > 50 || (inByteOld[i] - inByte[i]) < -50)
  {
    inByteOld[i]=inByte[i];
  }
  else
  {
    inByte[i]=inByteOld[i];
  }
}

I am not sure if that is the correct way to do it tho, just wanted some help on this

thanks!

update moderator: added code tags

Italics.
I really hate italics.

Quick_questions:
I am getting really ugly signals that just go up and down, but only in small differences, so what I wanted to do is to tell the sketch "only change the value if it is over a +/- 50 threshold"

One option is to smooth the incoming signal. An exponential decaying average is very easy to code and quite effective for things like this. The algorithm is:

smoothed = ((weight * smoothed) + newSample) / (weight + 1);

I'm sure you get the idea.

The other option is to record the current filtered value, and only change it when the new value is sufficiently far from it. For example:

if(abs(filtered - newValue) > threshold)
{
    filtered = newValue;
}
inByte[0] = float(inputStringArr[0])-600; 
 inByte[1] = float(inputStringArr[1])-600;
 inByte[2] = float(inputStringArr[2])-600;
 inByte[3] = float(inputStringArr[3])-600;
 inByte[4] = float(inputStringArr[4])-600;

What does the sending code look like? Why are you adding 600 to the value before converting it to a string?

On every call to serialEvent(), you are making a copy of the values in inByte in the new array inByteOld. There is nothing there to compare the new values to, so if the value is greater than 50, the difference will be too.

inByteOld needs to be global.

PaulS:

inByte[0] = float(inputStringArr[0])-600; 

inByte[1] = float(inputStringArr[1])-600;
inByte[2] = float(inputStringArr[2])-600;
inByte[3] = float(inputStringArr[3])-600;
inByte[4] = float(inputStringArr[4])-600;



What does the sending code look like? Why are you adding 600 to the value before converting it to a string?

On every call to serialEvent(), you are making a copy of the values in inByte in the new array inByteOld. There is nothing there to compare the new values to, so if the value is greater than 50, the difference will be too.

inByteOld needs to be global.

the sending code looks like

893 751 981 943 1023
878 741 981 943 1023
866 767 981 943 1023
887 748 981 943 1023
869 736 981 943 1023
906 760 981 943 1023
876 740 981 943 1023
888 748 981 943 1023
888 749 980 943 1023
897 755 980 943 1023
858 760 980 943 1023
871 737 979 943 1023
897 754 984 940 1023
. . ETC

I am subtracting 600 from each value since all the numbers are in the +700 range and my graphing window is 800x600 (so if I dont substract the values the whole screen is colored). since all the values oscilate above 600, thats why I subtracted 600.
(with commas instead of spaces)
so far I only have the first 2 channels connected (thats why the last 3 dont really move)

thanks, i'll try the global variable.

PeterH:

Quick_questions:
I am getting really ugly signals that just go up and down, but only in small differences, so what I wanted to do is to tell the sketch "only change the value if it is over a +/- 50 threshold"

One option is to smooth the incoming signal. An exponential decaying average is very easy to code and quite effective for things like this. The algorithm is:

smoothed = ((weight * smoothed) + newSample) / (weight + 1);

I'm sure you get the idea.

actually I dont get the idea, could you elaborate?

How about : take the current running total, multiply by seven eighths and add one eighth of the most recent sample?
(Substitute any unity sum fraction and timescale you wish)

the sending code looks like

Really? I find that hard to believe.

PaulS:

the sending code looks like

Really? I find that hard to believe.

Im assuming you mean the code sent from the arduino? (the serial monitor output)
or what did you mean by "the sending code"

what did you mean by "the sending code"

The Arduino is NOT sending code. It is executing some code that is sending data. We need to see that code to see HOW it is sending data. You can NOT expect to be able to deal with the data that is sent if you don't know what is being sent.

void setup()
{
  Serial.begin(115200); //setup serial
}

void loop()
{
  // Declare variables that will store values read from each sensor
int channels[5];

  //read input pins with analog values
  channels[0] = analogRead(A0);
  channels[1] = analogRead(A1);
  channels[2] = analogRead(A2);
  channels[3]= analogRead(A3);
  channels[4] = analogRead(A4);
  //print value on the serial monitor screen
  Serial.print(channels[0]);
  Serial.print(","); Serial.print(channels[1]);
  Serial.print(","); Serial.print(channels[2]);
  Serial.print(","); Serial.print(channels[3]);
  Serial.print(",");  Serial.print(channels[4]);
  
  Serial.println(""); //line break
  
  delay(200);
  
}

I already showed you the output data (the serial monitor)
all separated by commas and in the processing sketch I split them by commas again, I do get graphic output so I know its correcty, but I want to know how to make it "smooth"

I was reading over in wikipedia exponential decaying and I kind of understood it, however I dont know how to convert the concepts into code.

I've read some codes Here but when they list examples they all have like "pre-set" values, they dont include a streaming set of values (like serialevents)
thats why Im confused, I tried declaring the variable globally, but still its value should be declared inside the serial event shouldnt it? (I am assuming the code is read top-to-bottom?) therefore you cant declare values of variables that havent been stated (for exmaple if I declare the value of the global variable to equal the values read by the arduino, which havent been read since the serialevent hasnt even started)

I am not too sure about this last part tho.

should that IF structure be in some other function thats not in the serial event?

Quick_questions:

smoothed = ((weight * smoothed) + newSample) / (weight + 1);

I'm sure you get the idea.

actually I dont get the idea, could you elaborate?
[/quote]

The idea is that the input is a sequence of values which are varying slightly. The output is a value which represents the average of recent values so that it only changes relatively slowly. In the code I posted, smoothed is the output value and newSample is the new value. Weight determines how slowly the smoothed value changes. A weight of zero means the smoothed value is always equal to the newValue - no smoothing at all. Bigger numbers mean more smoothing - the smoothed value will change more slowly. If you don't get how it achieves this then make up a sequence of input numbers and run the calculation through in your head to see what it does.

PeterH:

Quick_questions:

smoothed = ((weight * smoothed) + newSample) / (weight + 1);

I'm sure you get the idea.

actually I dont get the idea, could you elaborate?

The idea is that the input is a sequence of values which are varying slightly. The output is a value which represents the average of recent values so that it only changes relatively slowly. In the code I posted, smoothed is the output value and newSample is the new value. Weight determines how slowly the smoothed value changes. A weight of zero means the smoothed value is always equal to the newValue - no smoothing at all. Bigger numbers mean more smoothing - the smoothed value will change more slowly. If you don't get how it achieves this then make up a sequence of input numbers and run the calculation through in your head to see what it does.
[/quote]

I get it, I understand the concept, again I dont know how to conver it from "concepts" to codes that work in my particular case, I tried this (similar to your example, I got it from wikipedia)

 import processing.serial.*;
 
 Serial myPort;        // The serial port
 int xPos = 1;         // horizontal position of the graph
 float [] Smooth0 = new float[5];
 float [] Smooth1 = new float[5];
 float [] inByte = new float[5];
 float [] inByte1 = new float[5];
 void setup () {
 // set the window size:
 size(800, 600);   
 myPort = new Serial(this, "COM4", 115200);
 // don't generate a serialEvent() unless you get a newline character:
 myPort.bufferUntil('\n');
 // set inital background:
 background(0);
 }
 void draw () {
 // everything happens in the serialEvent()
 }
 
 void serialEvent (Serial myPort)
 {
 // get the ASCII string:

 String inString = myPort.readStringUntil('\n');
 
 if (inString != null && inString.length() > 0) {
 // trim off any whitespace:
 inString = trim(inString);
 String [] inputStringArr = split(inString, ",");
 
 // convert to an int and map to the screen height:
 
 inByte1[0] = float(inputStringArr[0])-500; 
 inByte1[1] = float(inputStringArr[1])-500;
 inByte1[2] = float(inputStringArr[2])-500;
 inByte1[3] = float(inputStringArr[3])-500;
 inByte1[4] = float(inputStringArr[4])-500;
 
 // FILTER VALUES TO SMOOTHEN LINE USING SIMPLE MOVING AVERAGE

 inByte[0] = inByte1[0] + ((inByte1[0]-inByte[0])/2);
 inByte[1] = inByte1[1] + ((inByte1[1]-inByte[1])/2);
 inByte[1] = inByte1[2] + ((inByte1[2]-inByte[2])/2);
 inByte[1] = inByte1[3] + ((inByte1[3]-inByte[3])/2);
 inByte[1] = inByte1[4] + ((inByte1[4]-inByte[4])/2);
 
 inByte[0] = map(inByte[0], 0, 1023, 0, height);
 inByte[1] = map(inByte[1], 0, 1023, 0, height);
 inByte[2] = map(inByte[2], 0, 1023, 0, height);
 inByte[3] = map(inByte[3], 0, 1023, 0, height);
 inByte[4] = map(inByte[4], 0, 1023, 0, height);
 }

I declared the variables outside, however the values are declared inside the serialevent, but with those particular commands, I get crazier things. so I dont know whats up =/


thank you for the replies BTW

What are you reading from, on the Arduino?

If the values on the Arduino end are in the range 0 to 1023, and you subtract 500 on the Processing side, doesn't that make the range 0 to 523? Why are you then mapping from 0 to 1023 to 0 to height?

Perhaps you should get the handling of data working the way you want using ONE channel, then try to apply it to all 5 channels.

It isn't clear from your pictures exactly which color corresponds to what channel, or color is even relevant to channel. Get rid of 80% of the crap from the picture, to make it clear.

the picture is only showing 2 channels, I think its pretty clear that both channels exhibit noise in the bottom (unfiltered) and even more noise on the upper part (filtered). I noted earlier that I am only doing 2 channels for now. (purple is one, red is the other).

As for removing 80% of the picture, I am not sure what 80% do you mean. the screenshot of the code is only to show the difference between each code (and its output), the other part of the code was copy/pasted into this thread earlier, I only wanted to show the difference (which I clearly highlighted IMO)

The serial output and code from the arduino was also copy/pasted from their source earlier in this picture.

And as mentioned there is only 2 channels, and they are pretty clear and different colors, even though I understand why working only 1 channel is beneficial, working 2 channels does not make it any more confusing since its only the same arithmetic operation just copy-pasted into a different channel.

Also, as I mentioned Earlier I am subtracting 500 from the data since, as you said arduino ranges from 0-1023, HOWEVER, all of my values are above 600, I havent seen a single value drop below 600 (rarely gets to 700 actually), and If I leave it as is, I get the following graph:

NOTE: THIS IS ONLY 2 CHANNELS WITHOUT SUBTRACTING 500 FROM EACH VALUE:
as you can see in the code, it only draws 2 lines.

As you can see the graph is almost at the top of the screen, meaning if I ever got a value over 900 it would pretty much just color the whole screen without really giving me any useful data.

As for removing 80% of the picture, I am not sure what 80% do you mean.

The 80% that relates to channels 1, 2, 3, and 4. Leave only the 20% dealing with channel 0.

As you can see the graph is almost at the top of the screen, meaning if I ever got a value over 900 it would pretty much just color the whole screen without really giving me any useful data.

Suppose that you send a value, 750, that is in the range 0 to 1023.

You subract (or used to) 500 from that, giving a value of 250. Then, you pretend, when you do that mapping, that the value is still in the range 0 to 1023. It isn't. The value is in the range 0 to 523. You are introducing distortion right there.

If you want the plot to cover only 3/4 of the screen (why, I can't imagine), then change the to range to 0 to height * 0.75.

I still think it's better to deal with one channel, until you get that working the way you want. It seems to me that you want to draw a line for every sample. It seems that you want the length of the line to not change more than some amount, regardless of what the data shows as a change. Again, why you want to do this is not clear.

What is also not clear is what the Arduino is measuring on the analog pins.

PaulS:
Suppose that you send a value, 750, that is in the range 0 to 1023.

You subract (or used to) 500 from that, giving a value of 250. Then, you pretend, when you do that mapping, that the value is still in the range 0 to 1023. It isn't. The value is in the range 0 to 523. You are introducing distortion right there.

If you want the plot to cover only 3/4 of the screen (why, I can't imagine), then change the to range to 0 to height * 0.75.

I still think it's better to deal with one channel, until you get that working the way you want. It seems to me that you want to draw a line for every sample. It seems that you want the length of the line to not change more than some amount, regardless of what the data shows as a change. Again, why you want to do this is not clear.

What is also not clear is what the Arduino is measuring on the analog pins.

I will do it without subtracting the 500 then, no tampering with the results then. I'll just re-draw a bigger screen.

The arduino is connected to photodiodes (a pulse oxymeter to be exact), and those are their readings. I want to make it similar to this:

(this is a professional pulse oxymeter reading, with all the filters and digital filters needed):

Thats what I want to get. I want to measure big changes in the pulse since I am using that in my research. the oxymeter is in the knuckle, and I want to map out the change in pulse (or blood flow) as you move the finger.

NOTE I cant use a regular pulse oxymeter since they are fitted for the tip of the finger or the ear or the penis, but not for the knuckles.

I will do it without subtracting the 500 then, no tampering with the results then.

There isn't any problem with subtracting 500 from the value. Just change the from range in the map() call to match what you are actually mapping FROM.

My guess is that the professional chart you show is based on measuring values from the pulse oxymeter far more frequently than you are, and not converting those values to strings, sending them over a serial port, and converting them back to floats. I'm guessing, too, that there is some smoothing going on in hardware, not software. Finally, I'm guessing that the professional chart is using a sensor for more accurate than what you can afford.

But, I could be all wrong.

PaulS:

I will do it without subtracting the 500 then, no tampering with the results then.

There isn't any problem with subtracting 500 from the value. Just change the from range in the map() call to match what you are actually mapping FROM.

My guess is that the professional chart you show is based on measuring values from the pulse oxymeter far more frequently than you are, and not converting those values to strings, sending them over a serial port, and converting them back to floats. I'm guessing, too, that there is some smoothing going on in hardware, not software. Finally, I'm guessing that the professional chart is using a sensor for more accurate than what you can afford.

But, I could be all wrong.

I will try a different sampling rate, thanks for the advice,
is there a way to send values without converting to strings? just as raw data?

Semi-Professional oxymeters (hospital-quality) cost less than 20 dlls to manufacture, it literally is just a photodiode and an infra-red LED. probalby R&D oxymeters are higher in price, but commercial ones are capable of the graphs listed above.

and I do have filters in my circuit (hardware circuits), however, given the amount of cables in the circuit itself (the filters are separated from the arduino board, they are in their own board), I assume the cables pick up a lot of noise (AFAIK cables act like antennas and thats why there is so much noise in any cable-transmitted-signal).

What I just want is a code to smoothen that out, I will try playing w the sampling rate for now

is there a way to send values without converting to strings? just as raw data?

Yes. A n int is two bytes.

int sensorVal = analogRead(sensorPin);
Serial.write((byte *)sensorVal, 2);

Just be sure to change the way you read the data on the other end.

If you want to smooth out the peaks and valleys, you take the average or median of a group of data . Cycle through the entire data set.

You can then stretch and shift the visualization however you want through addition, subtraction, multiplication, etc...like how you would manipulate a sine curve.