HTTP Get not working anymore on MKR1000

I have a MKR1000 board which used to work fine last year.
Now I was about to improve its functionality, but it looks like the HTTP requests are not working anymore.

Here are some details which could be useful:

I’m using Arduino 1.8.0 on Windows 10.
This is printed in the serial monitor after a complete loop:

Attempting to connect to SSID: xray1
Connected to wifi
SSID: xray1
IP Address: 192.168.100.12
signal strength (RSSI):-47 dBm

Starting connection to server...
connected to server

disconnecting from server.

The code tested is taken from an example from the IDE:

/*
  Web client

 This sketch connects to a website (http://www.google.com)
 using a WiFi shield.

 This example is written for a network using WPA encryption. For
 WEP or WPA, change the WiFi.begin() call accordingly.

 This example is written for a network using WPA encryption. For
 WEP or WPA, change the WiFi.begin() call accordingly.

 Circuit:
 * WiFi shield attached

 created 13 July 2010
 by dlf (Metodo2 srl)
 modified 31 May 2012
 by Tom Igoe
 */


#include <SPI.h>
#include <WiFi101.h>
#include "arduino_secrets.h" 
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
char ssid[] = SECRET_SSID;        // your network SSID (name)
char pass[] = SECRET_PASS;    // your network password (use for WPA, or use as key for WEP)
int keyIndex = 0;            // your network key Index number (needed only for WEP)

int status = WL_IDLE_STATUS;
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(74,125,232,128);  // numeric IP for Google (no DNS)
char server[] = "www.google.com";    // name address for Google (using DNS)

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
WiFiClient client;

void setup() {
  //Initialize serial and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // check for the presence of the shield:
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    // don't continue:
    while (true);
  }

  // attempt to connect to WiFi network:
  while (status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);

    // wait 10 seconds for connection:
    delay(10000);
  }
  Serial.println("Connected to wifi");
  printWiFiStatus();

  Serial.println("\nStarting connection to server...");
  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected to server");
    // Make a HTTP request:
    client.println("GET /search?q=arduino HTTP/1.1");
    client.println("Host: www.google.com");
    client.println("Connection: close");
    client.println();
  }
}

void loop() {
  // if there are incoming bytes available
  // from the server, read them and print them:
  while (client.available()) {
    char c = client.read();
    Serial.write(c);
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting from server.");
    client.stop();

    // do nothing forevermore:
    while (true);
  }
}


void printWiFiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
}

I also tried other HTTP libraries with similar results.
Any ideas of why could have stopped working?

may be google stopped answering to requests that do not seem to come from a regular browser to avoid bots generating fake clicks for advertising

have you tried with a different URL, may be a web site you can control and are sure it's answering

What do you think the header "Connection: close" does?

Danois90:
What do you think the header "Connection: close" does?

what's your point? why would you want to keep it alive?

J-M-L:
...have you tried with a different URL, may be a web site you can control and are sure it's answering

Yes, same result.
As a matter of fact, I've tried the same URL which used to work.

do you see the request coming in on your server ? you did not change anything on the HTTP server side ?

It just seems to make no sense that the only purpose of the device is to perform one request and "forevermore" just waste energy. Anyhow, the request is malformed; "Accept" and "User-Agent" are required fields by most servers these days and "client.available()" may be true even if "client.connected()" is not.

Danois90:
It just seems to make no sense that the only purpose of the device is to perform one request and "forevermore" just waste energy. Anyhow, the request is malformed; "Accept" and "User-Agent" are required fields by most servers these days and "client.available()" may be true even if "client.connected()" is not.

All good points, but still does not explain what you were suggesting... It feels natural to close the connection once the (useless) request has been served.

@george_i --> I've seen indeed Servers being more demanding in terms of headers and requiring at least a User-Agent

J-M-L:
@george_i --> I've seen indeed Servers being more demanding in terms of headers and requiring at least a User-Agent

I tried a few different servers.
My server, however, had no change in the past year.
Maybe the libraries are now a bit different.

What seems to be is a connection time out.

Did you add the headers ?

J-M-L:
Did you add the headers ?

No. Not on the http request made from Arduino and neither on my server.

I actually doubt that it is related to headers.
If the server would've rejected the request because of the server I should see a rejection message.

But instead it hangs around 40-60 seconds and then it says "disconnected", which in fact means that is not connect, probably at all.

Rather than guessing it would be easier to check

A Server or component along the way can bluntly ignore you if they think it’s a denial of service attack and not even answer anything, leading to timeout

Can you check if your module has really internet access ? May be a NTP example

george_i:
But instead it hangs around 40-60 seconds and then it says "disconnected"

Sounds like the server is waiting for the request to complete but it does not complete before the connection times out - this would explain the behaviour and the lack of a response.

After a few more tests the situation so unchanged.
No change if I add the headers.
I also tried to read a page from the local network, with no luck.

I opened a ticket on the HTTP library, on GitHub.

If your device worked until you changed something, the libraries used are probably not to blame.

Danois90:
If your device worked until you changed something, the libraries used are probably not to blame.

The only thing I changed is the IDE along with the latest libraries versions.

So in August 2019 it was working fine.
Last week I installed the latest version of Arduino IDE and uploaded the code again.
But when I did, the new libraries versions were newer than I had last year.

Go back to the previous version and test again?

J-M-L:
Go back to the previous version and test again?

I wish, but I don't know what was before.

But why is not working even the simplest http get working now?
Strange

What wifi chip does your board use? That code goes back a while.

So I wanted to test for myself.

Here is what I did:

I have an web server hosted somewhere on the internet and so I created a “MKR” folder in the web root directory.

I created two very simple files

mkr.htm encoded in UTF8 with CR/LF as end of lines

<html>
<head>
<title>
MKR TEST PAGE
</title>
</head>
<body>
<p>This is a very simple HTML Page</p>
<p>to show I can reach out</p>
</body>
</html>

and a PHP script
mkr.php

<?php
echo "MKR GET REQUEST IS OK";
?>

I wrote the following code, which is very close to the example you have seen and uploaded it on a MKR1000

#include <WiFi101.h>

/* YOUR PERSONAL SETTINGS HERE */
const char* ssid      = "xxx";
const char* password  = "xxx";
const char* server    = "www.mydommain.com";

const char* request   = "GET /MKR/mkr.htm HTTP/1.1"; // or "GET /MKR/mkr.php HTTP/1.1"

int status = WL_IDLE_STATUS;
WiFiClient client;

#define DEBUG

#ifdef DEBUG
#define D_print(...)    Serial.print(__VA_ARGS__); client.print(__VA_ARGS__)
#define D_println(...)  Serial.println(__VA_ARGS__); client.println(__VA_ARGS__)
#else
#define D_print(...)    client.print(__VA_ARGS__)
#define D_println(...)  client.println(__VA_ARGS__)
#endif

void setup()
{
  Serial.begin(115200);
  while (!Serial);

  Serial.println(F("Connecting to Wi-Fi"));
  while (status != WL_CONNECTED) {
    status = WiFi.begin(ssid, password);
    if (status != WL_CONNECTED) {
      Serial.write('.');
      delay(1000);
    }
  }

  Serial.println(F("\nConnected"));
  Serial.print(F("IP : "));  Serial.println((IPAddress) WiFi.localIP());
  Serial.print(F("RSSI : ")); Serial.print(WiFi.RSSI()); Serial.println(F(" dBm"));

  Serial.println(F("\nSending Request"));
  if (client.connect(server, 80)) {
    D_println(request);
    D_print(F("Host: ")); D_println(server);
    D_println(F("User-Agent: Arduino MKR"));
    D_println(F("Connection: close")); // or keep-alive
    D_println(); // empty line to end the HTTP request
  } else {
    Serial.println(F("Could not connect to Server"));
  }
}


void loop()
{
  while (client.available())
    Serial.write((char) client.read());

  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting from server.");
    client.stop();
    while (true);
  }
}

The D_print macros basically send the data both to the Serial console opened at 115200 bauds and to the client. if you don’t define the DEBUG symbol the request only goes to the client.

You obviously need to provide your Wi-Fi information and web server info there

const char* ssid      = "xxx";
const char* password  = "xxx";
const char* server    = "www.mydommain.com";

I ran this code twice

1/ first with a request for the HTML page. The Serial Monitor (@ 115200 bauds) got me

[color=purple]Connecting to Wi-Fi
.
Connected
IP : 10.0.0.39
RSSI : -45 dBm

Sending Request
[color=blue]GET /MKR/mkr.htm HTTP/1.1
Host:[i]<snip>[/i]
User-Agent: Arduino MKR
Connection: close[/color]

HTTP/1.1 200 OK
Date: Sat, 02 May 2020 10:26:16 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 157
Connection: close
Server: Apache
Accept-Ranges: bytes
Vary: Accept-Encoding
Set-Cookie: SERVERID14=1330224|Yq1Pz|Zt1R1; path=/
X-IPLB-Instance: 39294

[color=red]<html>
<head>
<title>
MKR TEST PAGE
</title>
</head>
<body>
<p>This is a very simple HTML Page</p>
<p>to show I can reach out</p>
</body>
</html>[/color]

disconnecting from server.

[/color]

So I do get the web page back with the right headers.


2/Then I ran the same code but with a request for the php script

const char* request   = "GET /MKR/mkr.php HTTP/1.1";

and got in the Serial monitor

[color=purple]Connecting to Wi-Fi
.
Connected
IP : 10.0.0.39
RSSI : -45 dBm

Sending Request
[color=green]GET /MKR/mkr.php HTTP/1.1
Host:[i]<snip>[/i]
User-Agent: Arduino MKR
Connection: close[/color]

HTTP/1.1 200 OK
Date: Sat, 02 May 2020 10:29:39 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Server: Apache
X-Powered-By: PHP/7.0
Vary: Accept-Encoding
Set-Cookie: SERVERID14=1330224|Yq1Pz|Zt1R1; path=/
X-IPLB-Instance: 29655

15
[color=red]MKR GET REQUEST IS OK
[/color]0


disconnecting from server.

[/color]

So the PHP script executed fine as well and I got the answer back.

Finally I took as well a MKR1010 and just replaced

#include <WiFi101.h>

with

#include <WiFiNINA.h>

and ran both tests again and they worked in the same way.

(interestingly the RSSI on the MKR1010 was -38 dBm and was slow to connect whereas I was getting -45 dBm with the MKR1000)

So I don’t think there is an issue with the library. You probably need to explore more what’s going on in your network and along the way to your server.