Proper GET upload protocol to Weather Underground?

My Arduino Mega-based weather station has been working reliably 24/7 for over a year now, uploading readings every minute to a MySQL DB I have set up on a Raspberry Pi on my network. So I decided to start uploading the same readings to Weather Underground, so others could use them as well.

I followed the GET protocol as explained on WU's Wiki, and it worked fine for around a week. Then all of a sudden, the WU server stopped accepting readings, and no amount of reboots of my station, router or modem, or tweaks to the code to fix possible issues, would fix it. Nor was there likely a problem with WU's servers, as other local stations continued to upload readings regularly.

I tried to email WU but got no response. So I did a little googling, and saw that I might have been using the wrong GET format (even though it worked for a while). I tried using a different format, and voila, I was once again able to upload readings to WU, and have been ever since. This was some days ago.

Here's the original GET format I used:

GET https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=stationid&PASSWORD=password&dateutc=2017-03-01+13%3A42%3A00&tempf=52.92&temp2f=53.60&humidity=99.90&dewptf=52.90&baromin=29.78&softwaretype=ArduinoCustomBuilt&action=updateraw HTTP/1.1

Recv 277 bytes

SEND OK

+IPD,132:HTTP/1.1 200 OK
Content-type: text/html
Date: Wed, 01 Mar 2017 13:42:03 GMT
Content-Length: 8
Connection: keep-alive

success

And here's the updated GET format that now works:

GET /weatherstation/updateweatherstation.php?ID=stationid&PASSWORD=password&dateutc=2017-03-04+13%3A28%3A00&tempf=24.13&temp2f=24.98&humidity=41.60&dewptf=3.93&baromin=30.55&softwaretype=ArduinoCustomBuilt&action=updateraw HTTP/1.1
Host: https://weatherstation.wunderground.com
Connection: close

Recv 303 bytes

SEND OK

+IPD,127:HTTP/1.0 200 OK
Content-type: text/html
Date: Sat, 04 Mar 2017 13:28:03 GMT
Content-Length: 8
Connection: close

success

Why did the first one work for a while, then stopped working? Could they have changed something in the PHP scripts on their servers, or updated some component of the server software, so that the old format no longer worked? Any idea what that might be?

Incidentally, I'm using an ESP8266 WiFi board to upload the readings.

Anyone? I realize that this is more of a hardware than software let alone internet programming site, but given that this is a networking forum and that many Arduino projects do connect to the internet or at least local networks these days, and that HTTP is one of the most common protocols used for communication, it seems like a pertinent question. I'm honestly curious as to why the first method worked for a while then stopped working. Have I accidentally come across a common hacking technique and thus the silence?

Thanks for your information. Many people who built an Arduino Weather Station with the code we can find on internet are now facing the same problem : upload doesn't work anymore.

This is the first working code I see. Could you post your complete code used to upload data to Wunderground ? Many people like me would appreciate.

For your question, I found out that it could be related to the Get structure. In the past, an incomplete/bad GET structure would work even if it's not according to standard structure. New wunderground servers now require a better GET structure.

Thanks for uploading your code.

J Guy

Hello!

It seems that they made changes in WU server - without announcing anything, of course...

They never answered to my email too... so, i do understand you very well :wink:

Until Feb 2017, GET function should be finish with

client.println();

The first days of Mar 2017, they changed something... so i had to changed it to

client.println("/ HTTP/1.0\r\n\r\n");

And... some hours or days after... they changed something again...
... and after a lot of tries...
... i found that the GET function works now fine if you finish it like this

client.println("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n");

Try it and inform us if it works for you...

Greetings!
dims

Thanks dims !

Yesterday I found a sparkfun sketch to upload data to their server. I noticed simple changes very similar to yours. In addition, they close the connection with the server. With these changes, I succeeded in uploading my data. I was sending multiple GET parameters and wanted to keep doing this way to avoid making a bigstring.

Here the changes I made :

client.print(WEBPAGE);
client.print(F("ID=XXXX"));
client.print(F("&PASSWORD=XXXX"));
client.print(F("&dateutc=now"));
client.print(F("&winddir="));
// and so on ....
client.println(F("&action=updateraw HTTP/1.1")); // print changed to println and HTTP/1.1 added at the end
client.println(F("Host: http://weatherstation.wunderground.com")); // new line added
client.println("Connection: close"); // new line added
client.println();

Thanks again for your code, it would have solved my problem.

J Guy

I'll try to post the complete code later, don't have time now, but yeah, changing it to "proper" GET format (which I've found is not easy to get decent documentation on online), by moving the host to its own line, adding the proper NLs & CRs (via "\n\r"), and properly closing the TCP connection, did the trick. WU has been useless on all this. I suspect that they have their reasons for not informing people or even admitting that they changed something on their servers. Perhaps it had to do with security, and they don't want to discuss it for fear of giving out information that could be used to compromise their servers. Just a guess.

On that note, ever since I opened up my LAMP server to the internet so I can remotely SSH to it and view current weather readings off a simple PHP-based web site I built, every now and then I get a very basic "GET /HTML..." type request from an unfamiliar IP address, that I'm guessing is some bot or spider trying to find holes in publicly accessible servers for potential exploitation. I wonder if WU was concerned about more serious and targeted attacks to their previous server scripts and thus hardened them by using proper GET protocol, but don't want to talk about it so no one knows.

Thanks for posting this! I thought I was going nuts. I use thingspeak and an ESP8266 and have been uploading it that way for over a year. It just stopped working. I am using ThingHTTP to do the GET request so I'll have to troubleshoot a little more.

I got a brief response back from WU telling me that around the time the old GET format stopped working, they switched their servers to AWS (Amazon Web Services), which I assume requires the proper GET format. No announcement or warning though, plus their help pages on composing a GET string uses the improper format that used to work but no longer does. I'll try to get them to fix that.

Dear friends,

Here is the code i use for sending data (Arduino uno & Wifi shield) to WU - it's working without any problems for weeks, again...

[edited message, according to habanero next comment]

In the first lines of the code, setting as global variables

char SERVER[] = "rtupdate.wunderground.com";           // Realtime update server - RapidFire
//char SERVER [] = "weatherstation.wunderground.com";  // Standard server 
char WEBPAGE [] = "GET /weatherstation/updateweatherstation.php?";

char ID [] = "YourPWS_ID"; // Here, you must change it to yours
char PASSWORD [] = "YourPWS_Password"; // Here, you must change it to yours

... and, as part of the loop code...

     if (client.connect(SERVER, 80)) {
     Serial.print(F("... Connected to server: "));
     Serial.print(SERVER);
     char c = client.read();
     Serial.print(F(", Server response: "));
     Serial.write(c); 
     Serial.println(F(""));
          
     Serial.println(F("... Sending DATA "));
     Serial.println(F(""));
    
     
     client.print(WEBPAGE); 
     client.print("ID=");
     client.print(ID);
     client.print("&PASSWORD=");
     client.print(PASSWORD);
     client.print("&dateutc=");
     client.print("now");    
     
     client.print("&tempf=");
     client.print(Tempf);
     client.print("&baromin=");
     client.print(PressIn);
     client.print("&dewptf=");
     client.print(dewpointf);
     client.print("&humidity=");
     client.print(Hum);
     client.print("&solarradiation=");
     client.print(SRadiation);
     
        
     client.print("&softwaretype=Arduino%20UNO%20version1&action=updateraw&realtime=1&rtfreq=30");   //Using Rapid Fire, sending data 1time every 30sec
     

     //Finishing the communication
    
     //client.println(); // Stopped working, Feb 2017
     //client.println("/ HTTP/1.0\r\n\r\n"); // Stopped working, first days of March 2017
     client.println("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n"); // Working since first days of March 2017
    
     Serial.println(F("... Server Response:"));

     while(client.connected()) {
      while (client.available()) {
        char c = client.read();
        Serial.write(c);
      }
    }
    

    client.flush();
    client.stop();
    
    
    }
    else {
    Serial.println(F("Connection failed"));
    char c = client.read();
    Serial.write(c);
    client.flush();
    client.stop();}

Using the console, you can see Server responses...

You get a message finishing with "success" when everything goes fine

I hope this could help you...

Greetings from Greece!
dims

dims1978,

Thanks, this is more or less what I do to get it to work, and how it changed in early March (not May, as you wrongly have in the code above, btw).

It would be helpful to others, though, if you could include the definitions of both SERVER and WEBPAGE. These are both specific to WU, so you wouldn't be revealing unique info that's personal to you.

I still can't get this to work with Thingspeak and their ThingHTTP app. Any suggestions?

ThingHTTP

You have right, habanero... i edited my last message with the code according to your remarks

dims

My weather station has been offline for ages. I finally got around to working on the code. This post was a huge help, I never would have figured out the new GET request format. Thank you!

I'm using a Moteino RFM69W (like Arduino with RFM69 radio) to get the wireless data from my Davis ISS station, then I'm uploading to Weather Underground.

My WU station is:

I've been having some problems uploading lately. I'm seem to be connecting to rtupdate.wunderground.com okay, but I'm not getting an the HTTP OK response like this:

HTTP/1.0 200 OK
Content-type: text/html
Date: Wed, 03 Jan 2018 11:20:51 GMT
Content-Length: 8

success

Anyone else have a similar issue?

I am. Occasionally I get an OK reply. Do you think they changed something again or it is just an infrastructure problem? When I use my web browser to send the HTTP request, sometimes I get an error too.

Has anyone managed to fix this issue?

I'm having similar issues to those that have posted above, but wondered if anyone had worked out a solution?

Yeap! Seems that WU changed smth again.
I cant update my PWS too

CrashEd:
Has anyone managed to fix this issue?

I'm having similar issues to those that have posted above, but wondered if anyone had worked out a solution?

Crash, try to check all your variables what they content. My wind and temperature was incorrect: 832 and 8123.

I fix that and try another way to update, you can try it. Code below.

void wunderground() {
  if ((millis() - wMillis) > 10000) {
    if (ethClient.connect("rtupdate.wunderground.com", 80)) {
      p=p.toFloat()*0.0296;  //metric to US
      dew=(dew * 1.8 +32);
      n=(n.toFloat()*10);
      ethClient.print("GET /weatherstation/updateweatherstation.php?ID=_YOUR_PWS_&PASSWORD=_YOUR_PASSWORD_&dateutc=now&tempf=");
      ethClient.print(tf);
      ethClient.print("&winddir=");
      ethClient.print(va);
      ethClient.print("&windspeedmph=");
      ethClient.print(w);
      ethClient.print("&windgustmph=");
      ethClient.print(windgustmph);
      ethClient.print("&baromin=");
      ethClient.print(p);
      ethClient.print("&dewptf=");
      ethClient.print(dew);
      ethClient.print("&humidity=");
      ethClient.print(h);
      ethClient.print("&UV=");
      ethClient.print(x);
      ethClient.print("&solarradiation=");
      ethClient.print(n);
      ethClient.print("&windspdmph_avg2m=");
      ethClient.print(windavg);
      ethClient.print("&softwaretype=Arduino&action=updateraw&realtime=1&rtfreq=10");
      ethClient.print("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n");
      ethClient.print("\r\n\r\n\r\n"); //just for to be sure =)
      ethClient.println();
      ethClient.println();
      ethClient.println();
      Serial.print("GET /weatherstation/updateweatherstation.php?ID=&PASSWORD=&dateutc=now&tempf=");
      Serial.print(tf); //starts here just check what you send
      Serial.print("&winddir=");
      Serial.print(va);
      Serial.print("&windspeedmph=");
      Serial.print(w);
      Serial.print("&windgustmph=");
      Serial.print(windgustmph);
      Serial.print("&baromin=");
      Serial.print(p);
      Serial.print("&dewptf=");
      Serial.print(dew);
      Serial.print("&humidity=");
      Serial.print(h);
      Serial.print("&UV=");
      Serial.print(x);
      Serial.print("&solarradiation=");
      Serial.print(n);
      Serial.print("&windspdmph_avg2m=");
      Serial.print(windavg);
      Serial.print("&softwaretype=Arduino&action=updateraw&realtime=1&rtfreq=10");
      Serial.print("/ HTTP/1.1\r\nHost: host:port\r\nConnection: close\r\n\r\n");
      Serial.print("\r\n\r\n\r\n");
      delay(200); //depends on your internet connection.
      ethClient.stop();
      Serial.println(F("transm ok")); //just to know that IF is ended. this is not checking
      wMillis = millis();
    }
  
  }
}

tyshkka:
Crash, try to check all your variables what they content. My wind and temperature was incorrect: 832 and 8123.

I fix that and try another way to update, you can try it. Code below.

Hi tyshkka,

Thanks - that's fixed it! One of the variables was incorrect!

Cheers,
'Crash