Go Down

Topic: Proper GET upload protocol to Weather Underground? (Read 2604 times) previous topic - next topic

habanero

Mar 10, 2017, 02:42 pm Last Edit: Mar 10, 2017, 03:01 pm by habanero
My Arduino Mega-based weather station has been working reliably 24/7 for over a year now, uploading readings every minute to a MySQL DB I have set up on a Raspberry Pi on my network. So I decided to start uploading the same readings to Weather Underground, so others could use them as well.

I followed the GET protocol as explained on WU's Wiki, and it worked fine for around a week. Then all of a sudden, the WU server stopped accepting readings, and no amount of reboots of my station, router or modem, or tweaks to the code to fix possible issues, would fix it. Nor was there likely a problem with WU's servers, as other local stations continued to upload readings regularly.

I tried to email WU but got no response. So I did a little googling, and saw that I might have been using the wrong GET format (even though it worked for a while). I tried using a different format, and voila, I was once again able to upload readings to WU, and have been ever since. This was some days ago.

Here's the original GET format I used:

Code: [Select]
GET https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=stationid&PASSWORD=password&dateutc=2017-03-01+13%3A42%3A00&tempf=52.92&temp2f=53.60&humidity=99.90&dewptf=52.90&baromin=29.78&softwaretype=ArduinoCustomBuilt&action=updateraw HTTP/1.1

Recv 277 bytes

SEND OK

+IPD,132:HTTP/1.1 200 OK
Content-type: text/html
Date: Wed, 01 Mar 2017 13:42:03 GMT
Content-Length: 8
Connection: keep-alive

success


And here's the updated GET format that now works:

Code: [Select]
GET /weatherstation/updateweatherstation.php?ID=stationid&PASSWORD=password&dateutc=2017-03-04+13%3A28%3A00&tempf=24.13&temp2f=24.98&humidity=41.60&dewptf=3.93&baromin=30.55&softwaretype=ArduinoCustomBuilt&action=updateraw HTTP/1.1
Host: https://weatherstation.wunderground.com
Connection: close

Recv 303 bytes

SEND OK

+IPD,127:HTTP/1.0 200 OK
Content-type: text/html
Date: Sat, 04 Mar 2017 13:28:03 GMT
Content-Length: 8
Connection: close

success


Why did the first one work for a while, then stopped working? Could they have changed something in the PHP scripts on their servers, or updated some component of the server software, so that the old format no longer worked? Any idea what that might be?

Incidentally, I'm using an ESP8266 WiFi board to upload the readings.

habanero

Anyone? I realize that this is more of a hardware than software let alone internet programming site, but given that this is a networking forum and that many Arduino projects do connect to the internet or at least local networks these days, and that HTTP is one of the most common protocols used for communication, it seems like a pertinent question. I'm honestly curious as to why the first method worked for a while then stopped working. Have I accidentally come across a common hacking technique and thus the silence?

jgveill

Thanks for your information. Many people who built an Arduino Weather Station with the code we can find on internet are now facing the same problem : upload doesn't work anymore.

This is the first working code I see. Could you post your complete code used to upload data to Wunderground ? Many people like me would appreciate.

For your question, I found out that it could be related to the Get structure. In the past, an incomplete/bad GET structure would work even if it's not according to standard structure.  New wunderground servers now require a better GET structure.

Thanks for uploading your code.

J Guy


dims1978

#3
Mar 21, 2017, 08:04 pm Last Edit: Mar 21, 2017, 08:24 pm by dims1978 Reason: typo
Hello!

It seems that they made changes in WU server - without announcing anything, of course...

They never answered to my email too... so, i do understand you very well ;-)

Until Feb 2017, GET function should be finish with

Code: [Select]
client.println();

The first days of Mar 2017, they changed something... so i had to changed it to

Code: [Select]
client.println("/ HTTP/1.0\r\n\r\n");

And... some hours or days after... they changed something again...
... and after a lot of tries...
... i found that the GET function works now fine if you finish it like this

Code: [Select]
client.println("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n");

Try it and inform us if it works for you...

Greetings!
dims

jgveill

Thanks dims !

Yesterday I found a sparkfun sketch to upload data to their server.  I noticed simple changes very similar to yours.  In addition, they close the connection with the server.  With these changes, I succeeded in uploading my data. I was sending multiple GET parameters and wanted to keep doing this way to avoid making a bigstring.

Here the changes I made :

    client.print(WEBPAGE);         
    client.print(F("ID=XXXX")); 
    client.print(F("&PASSWORD=XXXX"));
    client.print(F("&dateutc=now"));         
    client.print(F("&winddir="));
   // and so on ....
    client.println(F("&action=updateraw HTTP/1.1"));   // print changed to println and HTTP/1.1               added at the end
    client.println(F("Host: http://weatherstation.wunderground.com")); // new line added
    client.println("Connection: close");   // new line added
    client.println();   

Thanks again for your code, it would have solved my problem.

J Guy

habanero

I'll try to post the complete code later, don't have time now, but yeah, changing it to "proper" GET format (which I've found is not easy to get decent documentation on online), by moving the host to its own line, adding the proper NLs & CRs (via "\n\r"), and properly closing the TCP connection, did the trick. WU has been useless on all this. I suspect that they have their reasons for not informing people or even admitting that they changed something on their servers. Perhaps it had to do with security, and they don't want to discuss it for fear of giving out information that could be used to compromise their servers. Just a guess.

On that note, ever since I opened up my LAMP server to the internet so I can remotely SSH to it and view current weather readings off a simple PHP-based web site I built, every now and then I get a very basic "GET /HTML..." type request from an unfamiliar IP address, that I'm guessing is some bot or spider trying to find holes in publicly accessible servers for potential exploitation. I wonder if WU was concerned about more serious and targeted attacks to their previous server scripts and thus hardened them by using proper GET protocol, but don't want to talk about it so no one knows.

nightshark

Thanks for posting this! I thought I was going nuts. I use thingspeak and an ESP8266 and have been uploading it that way for over a year. It just stopped working. I am using ThingHTTP to do the GET request so I'll have to troubleshoot a little more.

habanero

I got a brief response back from WU telling me that around the time the old GET format stopped working, they switched their servers to AWS (Amazon Web Services), which I assume requires the proper GET format. No announcement or warning though, plus their help pages on composing a GET string uses the improper format that used to work but no longer does. I'll try to get them to fix that.

dims1978

#8
Mar 30, 2017, 01:11 am Last Edit: Apr 01, 2017, 07:24 am by dims1978
Dear friends,

Here is the code i use for sending data (Arduino uno & Wifi shield)  to WU - it's working without any problems for weeks, again...

[edited message, according to habanero next comment]

 In the first lines of the code, setting as global variables

Code: [Select]
 
char SERVER[] = "rtupdate.wunderground.com";           // Realtime update server - RapidFire
//char SERVER [] = "weatherstation.wunderground.com";  // Standard server
char WEBPAGE [] = "GET /weatherstation/updateweatherstation.php?";

char ID [] = "YourPWS_ID"; // Here, you must change it to yours
char PASSWORD [] = "YourPWS_Password"; // Here, you must change it to yours

... and, as part of the loop code...

Code: [Select]
   
     if (client.connect(SERVER, 80)) {
     Serial.print(F("... Connected to server: "));
     Serial.print(SERVER);
     char c = client.read();
     Serial.print(F(", Server response: "));
     Serial.write(c);
     Serial.println(F(""));
         
     Serial.println(F("... Sending DATA "));
     Serial.println(F(""));
   
     
     client.print(WEBPAGE);
     client.print("ID=");
     client.print(ID);
     client.print("&PASSWORD=");
     client.print(PASSWORD);
     client.print("&dateutc=");
     client.print("now");   
     
     client.print("&tempf=");
     client.print(Tempf);
     client.print("&baromin=");
     client.print(PressIn);
     client.print("&dewptf=");
     client.print(dewpointf);
     client.print("&humidity=");
     client.print(Hum);
     client.print("&solarradiation=");
     client.print(SRadiation);
     
       
     client.print("&softwaretype=Arduino%20UNO%20version1&action=updateraw&realtime=1&rtfreq=30");   //Using Rapid Fire, sending data 1time every 30sec
     

     //Finishing the communication
   
     //client.println(); // Stopped working, Feb 2017
     //client.println("/ HTTP/1.0\r\n\r\n"); // Stopped working, first days of March 2017
     client.println("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n"); // Working since first days of March 2017
   
     Serial.println(F("... Server Response:"));

     while(client.connected()) {
      while (client.available()) {
        char c = client.read();
        Serial.write(c);
      }
    }
   

    client.flush();
    client.stop();
   
   
    }
    else {
    Serial.println(F("Connection failed"));
    char c = client.read();
    Serial.write(c);
    client.flush();
    client.stop();}

Using the console, you can see Server responses...

You get a message finishing with "success" when everything goes fine

I hope this could help you...

Greetings from Greece!
dims

habanero

dims1978,

Thanks, this is more or less what I do to get it to work, and how it changed in early March (not May, as you wrongly have in the code above, btw).

It would be helpful to others, though, if you could include the definitions of both SERVER and WEBPAGE. These are both specific to WU, so you wouldn't be revealing unique info that's personal to you.

nightshark

I still can't get this to work with Thingspeak and their ThingHTTP app. Any suggestions?

ThingHTTP

dims1978

You have right, habanero... i edited my last message with the code according to your remarks

dims

Go Up