Help required coding rules comparing received data to float and fixed values

I have adapted and written code for my home temperature project. It works a dream most of the time but sometimes the data is corrupted and I get bad readings.

10:00 2nd Floor data 24.0C 52%,
10:05 2nd Floor data -32.7C 111%,
10:10 2nd Floor data 24.0C 52%

13:45 3rd Floor data 24.4C 50%
13:50 3rd Floor data -25.0C 100%
13:55 3rd Floor data 24.4C 50%

I want to add two rules to my code that would eliminate all the obviously wrong readings I have captured so far.

The full code is here (too long to paste)

The two rules I want to add to the code are;

Do not use the reading as the float value IF:

• the humidity reading is over 100%
• the temperature reading is more than 10 degrees away from the current float value

With the humidity you can see that I have already limited bad readings where the humidity is over 128% by adapting the code for decoding the signal to look for a bit in the signal that is always 0. My attempts to take the reading value and compare it to a fixed value failed.
In relation to temperature the readings are received every minute so there should never be a very dramatic change. I found that trying to compare to a range of values that changes according to the current float value was beyond me.

Any help would be appreciated.

Thank you

PS I tried to crack the CRC but it has turned out to be more difficult than I thought.
PPS I should also point out that the number of bad readings is very small, the last batch of data contained 1,254 readings, of which only 4 were bad. Still it mucks up my graphs!

I think I'd be more interested in knowing why I'm getting bogus readings.

You can attach your code at the bottom of your post. I'm not taking the trouble to get it from Github.

The two tests you have proposed would be very simple to implement - but I am inclined to @econjack's viewpoint.

...R

Possible causes:

• When you use the data just as it is, some noise peaks might have altered the data.
• When you receive data packages without checksum, once in a while the RF noise will seem like a data package. Even with an 8-bit checksum, you might expect a data package once a month that is actually just noise. Ignoring RF noise is what has been done a lot, and causing a lot of trouble to those who did.
• It could be the sensor itself, sending wrong data once in a while. I hope it doesn't, but if it does, the checksum should prevent that it will be used.
• Your Arduino has TIMER0 running for millis() timing. That interrupt from TIMER0 might disturb the timing a little.

Possible solution:

• You have to use the RF 433MHz checksum, and you have to use the CRC of the data package from the device (or is there just one checksum ?). It is the only way to do it right.
• A quick and dirty solution is to remove bad data packages. You could have a buffer of 3 data packages, and if the middle one does not fit with the others, remove that data.

According to the original analysis of the transmissions by this sensor, each data packet is sent 3 times. Decoding RF signal from Ambient Weather Thermo-Hygrometer - Device Hacking - Arduino Forum

If you are now analyzing just one of the packets, consider analyzing all three and take the two that agree, or discard transmissions where they don't agree.

Thanks for all your ideas!

Reason for the bogus readings

As has already been suggested, the signals are broadcast on 434MHZ and picked up by a RX. The signal is subject to interference from neighbor's equipment using 434Mhz (I can detect data bursts from other sensors) as well as the structure of my house and electrical equipment. To deal with these physical problems I have undertaken a lot of experimentation with antenna material, length and unit placement.

I believe that interference when picking up 434Mhz is a common problem and why the data burst from the sensors includes the same data three times and includes a CRC to weed out the bad readings.

CRC analysis

The sensors are produced by Ambient Weather, I did not build them myself. I have not been able to crack the checksum and they are not able to provide the details to me (I asked them). I have asked for help on several forums and learnt a lot about CRC, but no success so far.

Comparing data packets

I tried comparing values between the three readings (there are several existing sketches that do this) but it did not work because:

• The first header is shorter than the second and third so often missed
• Some of the more distant sensors may only succeed in sending a single valid packet

I tried shortening the header that was being looked for by the sketch but it actually gave more bad readings. I tried comparing the two captured reading but some sensors stopped reporting as obviously bad data caused the good and bad data to be thrown away.

Eliminating bad data by analysis

My sketch already filters bad readings where there is a problem with the header, the sensor identifier, the channel number and humidity value. Here is an example from some binary data I collected earlier this month

Valid headers received: 3,912
Valid header and sensor ID received: 3,841
Valid header, sensor ID and channel number received: 3.838
Valid header, sensor ID, channel number and humidity < 128% received: 3,836

The number of bogus signals still reported by the sketch is now very low but by adding my proposed two rules I would eliminate all the bogus readings that the sketch still reports.

Thank you!

P.S. I have edited this post to include the correct ino file below

Working_with_analysis___errorck.ino (15.1 KB)

I've not worked with 434MHZ signals, but have some amateur radio experience. Would some kind of band pass filters be helpful?

I don't know whether it is because you uploaded a .txt rather than an .ino file, or whether it is to do with the editor you are using but the attached file is unusable because it has an extra line-feed between every line which makes it too difficult to read.

Your's is not the first case where this happened, but I'm sorry, I don't know the cause or solution.

Can you provide a general description of how the program is supposed to work?

...R

Sorry, ino file now attached!

A band pass filter may well help reduce the signal noise although I would prefer a software solution as the RX is a single unit and I am keen to keep the electronics (and number of components) to a minimum.

A working example of code to compare an input value with a float and fixed value would be great.

Thanks again,

Working_with_analysis___errorck.ino (15.1 KB)

I don't know the cause or solution.

It is probably because you are not using Windows.

It is even more annoying, to be a Windows user and to get files which have no line breaks in them at all.

At least you can read the ones you are getting.

Tops:
The sensors are produced by Ambient Weather, I did not build them myself. I have not been able to crack the checksum and they are not able to provide the details to me (I asked them). I have asked for help on several forums and learnt a lot about CRC, but no success so far.

Really? It took me about 10 seconds of Googling to find this...

The final 8 bits of the packet are a CRC-8 over the payload - i.e. over the four preceding bytes. (The preamble is not included in the CRC). The CRC-8 polynomial used is x^8 + x^5 + x^4 + 1.

Query...
https://www.google.com/search?q=Ambient+Weather+wireless+protocol+434mhz

Robin2:
...the attached file is unusable because it has an extra line-feed between every line...

@Robin2, you need to find yourself a better source code editor. I use Visual Studio, Delphi, and sometimes Programmer's Notepad (it comes with the Arduino IDE).

Tops:
Sorry, ino file now attached!

Thanks, that is much better.

But the bad news is that there is an awful lot of complicated code and you haven't provided any explanation.

I use Gedit and I'm very happy with it.
I think the stuff you mention is for Windows.

...R

Hey Coding badly,

Thank you for googling the ambient weather station, I never thought about that!?!?!?

I am familiar with the page you provided and it was helpful in hacking the packets, but you will notice that the one you googled is a WH2, mine is F007th.

On closer examination you will also see that the format of the data within the packet is different.

I have already tried using the same CRC (0x31) without success. I have even put the CRC data through reveng (for reverse engineering CRCs) without a match. On this occasion the 10 second google result has not helped answer my question.

It would be really helpful and much appreciated however if you could suggest a few lines of code to compare input values with floating values and fixed values!

Since the temperature and humidity usually don't change much from transmission to transmission, you could keep a running, low pass filtered average of each and then ask if the newest measurement deviates by more than some reasonable value from that average.

Each time through the loop, a running low pass filter for floating point or integer quantities might be calculated as follows (in both cases, old_value should be initialized elsewhere)

for a float

new_value = get_from_device();
//check for deviation of new from filtered here, if OK ...
filtered_value = 0.9old_value + 0.1new_value;
old_value = filtered_value;

for an integer

new_value = get_from_device();
//check for deviation of new from filtered here, if OK...
filtered_value = (7*old_value+new_value+4)/8; //rounded integer divide for positive quantities
old_value = filtered_value;

One check on validity might be whether the ratio of new_value to filtered_value is much different than 1. For that matter, such a test would probably work, just comparing the new and last measurements. Whatever you do, it will be empirical.

Thank you everybody, after a productive afternoon, I now have what I wanted coded (better annotated and explained as requested). This forum thread was invaluable;

1. Thank you Jremington for providing the example and making me realize the problem I had was;

• I did not have enough values to do the comparison I wanted,
• I was trying to do the comparison after setting the values rather than before.

2. Thank you nearly everybody else for focusing on my project and not my question. As a result I spent time thinking how I could rephrase the question without referring to my project and, combined with Jremington's reply, found my own answer.

Previous code

int stnId = ((manchester[0][3]&B01110000)/16)+1;  // looks at 3 bits in byte 3 used to identify channels 1 to 8
dataType = manchester[0][1];  // looks in byte 1 for the F007th Ambient Thermo-Hygrometer code (0x45)
 if ((dataType == 0x45) && (stnId == 1)){ //if the packet is from a F007th sensor on channel 1
  Ch1temp = (float(((manchester[0][3]&B00000111)*256)+ manchester[0][4])-720)*0.0556; // looks in bytes 3 and 4 for temperature and then convert to C
  Ch1hum =(manchester [0][5]); // looks in byte 5 for humidity data

New code with additional values "newtemp", "newhum", and "differencetemp"

int stnId = ((manchester[0][3]&B01110000)/16)+1;  // looks at 3 bits in byte 3 used to identify channels 1 to 8
dataType = manchester[0][1];  // looks in byte 1 for the F007th Ambient Thermo-Hygrometer code (0x45)
Newtemp = (float((((manchester[0][3]&B00000111)*256)+ manchester[0][4])-720)*0.0556); // looks in bytes 3 and 4 for temperature and then converts to C
Newhum =(manchester [0][5]); // looks in byte 5 for humidity data
   if ((dataType == 0x45) && (stnId == 1) && (Newhum <= 100)){ // if the packet is from a F007th sensor on channel 1 and humidity equal or less than 100
    if (Ch1temp == 100){ // if the channel 1 temperature is 100C (default when sketch started so first reading)
    Ch1temp = Newtemp; // take the new reading as the offical channel 1 temperature
    Ch1hum = Newhum; // take the new reading as the offical channel 1 humidity
    }
    if (Ch1temp != 100){ // if the channel 1 temperature is other than 100C(so a subsequent reading)
    differencetemp = Newtemp - Ch1temp; // subtract the previous reading from the new reading to find the difference
    if (differencetemp < 10 && differencetemp > -10){ // if the new reading is within 10 degrees of the old one
    Ch1temp = Newtemp; // take the new reading as the offical channel 1 temperature
    Ch1hum = Newhum;} // take the new reading as the offical channel 1 humidity

As you can see there are now two scenarios. I set the temperature of all the sensors to 100 initially (because this is a value that should not occur during normal operation!) When a packet is received the sketch checks it is a F007th sensor, the channel number and whether the humidity is 100 or less. Then it checks the temperature, if the channel temperature value is 100 the sketch simply records the temperature received in the packet (it is the first reading taken). However for all future packets it compares the new temperature against the stored value, ignoring the new value if it has changed by more than 10 degrees. As the packets are sent every minute, I think a change of temperature of more than 10 degrees in such a short time is very unlikely.

So that is how you improve a sketch to error check because, despite google, you cannot crack the CRC.

P.S. Yes I know, you have to manually check that the first reading is not one of the 0.3% of bad values. If it is then that reading will get stuck.

The new version is not much easier to read than this

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam ut mollis lectus. In ipsum nunc, fermentum id faucibus ac, ullamcorper sit amet metus. Phasellus varius tortor non mattis ornare. In pretium, sapien et dictum sodales, arcu orci ullamcorper libero, sed commodo tortor est ac ligula. Maecenas a lacus porttitor, lacinia libero quis, porta libero. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vivamus ipsum metus, tristique vitae nibh quis, vehicula ultricies elit. Suspendisse scelerisque sagittis risus, vehicula congue nisi ornare a. Suspendisse consequat tempor est ornare sollicitudin. Fusce maximus metus metus, nec malesuada justo rutrum quis. Donec vestibulum tempus lectus nec hendrerit. Sed ut orci blandit, pretium velit eget, ullamcorper urna.

Have you considered using whitespace to assist the eye (and the brain) to separate different elements.

...R

First, thank you to those who worked on the F007TH sketch, especially the manchester decoding.
I choose the Ambient for my project since the code existed, but was not happy not being able to verify the checksum. So I've reversed engineered it. The code is below and here is a link to an article on how I tackled it:

I hope that is payback to those who did the heavy lifting on the decoding.

uint8_t Checksum(int length, uint8_t *buff)
{
    uint8_t mask = 0x7C;
    uint8_t checksum = 0x64;
    uint8_t data;
    int byteCnt;	

    for ( byteCnt=0; byteCnt < length; byteCnt++)
    {
    	int bitCnt;
	data = buff[byteCnt];
	
	for ( bitCnt= 7; bitCnt >= 0 ; bitCnt-- )
	{
            uint8_t bit;
			
            // Rotate mask right
	    bit = mask & 1;
	    mask =  (mask >> 1 ) | (mask << 7);
	    if ( bit )
	    {
		mask ^= 0x18;
	    }
			
	    // XOR mask into checksum if data bit is 1			
	    if ( data & 0x80 )
	    {
	    	checksum ^= mask;
            }
	    data <<= 1;	
	}
    }
    return checksum;
}

@BaronVonSchnowzer, thank you for the follow-up!

FYI, the link does not work.

Thanks CodingBadly. Fixed the link.