WiFiS3 http / https GET issues

So I loaded in example code for a simple http GET request and it successfully in the serial monitor prints out google's site. That's all well in good.

Now I have this JSON object I have on my own website:

https://www.bryancolvin.com/bender/
which simply prints out:

{ "P1": 0, "P2": 0, "P3": 0, "P4": 0 }

In the browser, it successfully prints out a simple JSON output. But I can't seem to connect to this with any of the example code. I'm getting a 301 permanently moved response! But it's there!

I think this has to do with how hostgator (my cheap hosting company) is maybe masking things on shared hosting? OR.....

I also noticed it changes it to https. So I think technically the browser is following a redirect to http SSL type connection on 443 port.

So I tried the example code for " Wi-Fi® Web Client SSL". The example does a GET request to google server and it works fine.

Now I swap the host server name and GET request to be my website / page like so:

/*
  TLS WiFi Web client

  Board CA Root certificate bundle is embedded inside WiFi firmware:
  https://github.com/arduino/uno-r4-wifi-usb-bridge/blob/main/certificates/cacrt_all.pem

  Find the full UNO R4 WiFi Network documentation here:
  https://docs.arduino.cc/tutorials/uno-r4-wifi/wifi-examples#wi-fi-web-client-ssl
*/

#include "WiFiS3.h"
#include "WiFiSSLClient.h"
#include "IPAddress.h"

#include "arduino_secrets.h"

///////please enter your sensitive data in the Secret tab/arduino_secrets.h
char ssid[] = SECRET_SSID;        // your network SSID (name)
char pass[] = SECRET_PASS;        // your network password (use for WPA, or use as key for WEP)

int status = WL_IDLE_STATUS;
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(74,125,232,128);  // numeric IP for Google (no DNS)
char server[] = "www.bryancolvin.com";    // name address for Google (using DNS)

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
WiFiSSLClient client;

/* -------------------------------------------------------------------------- */
void setup() {
/* -------------------------------------------------------------------------- */
  //Initialize serial and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // check for the WiFi module:
  if (WiFi.status() == WL_NO_MODULE) {
    Serial.println("Communication with WiFi module failed!");
    // don't continue
    while (true);
  }

  String fv = WiFi.firmwareVersion();
  if (fv < WIFI_FIRMWARE_LATEST_VERSION) {
    Serial.println("Please upgrade the firmware");
  }

  // attempt to connect to WiFi network:
  while (status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network.
    status = WiFi.begin(ssid, pass);

    // wait 10 seconds for connection:
    delay(10000);
  }

  printWifiStatus();

  Serial.println("\nStarting connection to server...");
  // if you get a connection, report back via serial:

  if (client.connect(server, 443)) {
    Serial.println("connected to server");
    // Make a HTTP request:
    client.println("GET / HTTP/1.1");
    client.println("Host: www.bryancolvin.com");
    client.println("Connection: close");
    client.println();
  }
}

/* just wrap the received data up to 80 columns in the serial print*/
/* -------------------------------------------------------------------------- */
void read_response() {
/* -------------------------------------------------------------------------- */
  uint32_t received_data_num = 0;
  while (client.available()) {
    /* actual data reception */
    char c = client.read();
    /* print data to serial port */
    Serial.print(c);
    /* wrap data to 80 columns*/
    received_data_num++;
    if(received_data_num % 80 == 0) {
      Serial.println();
    }
  }
}

/* -------------------------------------------------------------------------- */
void loop() {
/* -------------------------------------------------------------------------- */
  read_response();

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting from server.");
    client.stop();

    // do nothing forevermore:
    while (true);
  }
}

/* -------------------------------------------------------------------------- */
void printWifiStatus() {
/* -------------------------------------------------------------------------- */
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your board's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
}

And the serial monitor prints out (copied it after my ip address):

signal strength (RSSI):-67 dBm

Starting connection to server...

disconnecting from server.

So... with my cheap host gator server... i think there's some funky redirecting/alt port ... something something, going on that I don't understand. The seemingly simple, load a JSON output is ending up difficult because of this hurdle. Oof.

I also tried curl in terminal (I'm on a mac)

and curl shows the 301, curl-L shows it follows the redirect and works.

curl in terminal gives:

bryancolvin@Bryans-MacBook-Pro ~ % curl http://bryancolvin.com/bender   
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.bryancolvin.com/bender">here</a>.</p>
</body></html>

so curl -L http://bryancolvin.com/bender gives this output:

bryancolvin@Bryans-MacBook-Pro ~ % curl -L http://bryancolvin.com/bender
{
  "P1": 0,
  "P2": 0,
  "P3": 0,
  "P4": 0
}

I'm sure I'm doing one thing wrong that's an easy fix though.... I'd be grateful to anyone that can shed light on this.

Maybe you aren't waiting long enough for a response.
The server will close the connection when it is finished sending.
This is what I would use:

while(client.connected()) {
  while (client.available()) {
    /* actual data reception */
    char c = client.read();
    /* print data to serial port */
    Serial.print(c);
    /* wrap data to 80 columns*/
    received_data_num++;
    if(received_data_num % 80 == 0) {
      Serial.println();
    }
  }
}

Tried that, it just sits there after server connection. In the example code it doesn't look like its making any funciton call to close the connection, the if (!client.connected()) seems to automatically update based on the inner workings of that class.

Thanks for the thought!

You're asking for www.bryancolvin.com/. With curl, you first tried http://bryancolvin.com/bender. The server did two standard things in the 301 redirect

  • use https -- very common to move folks over to a secure connection
  • use www -- less common, but to move you off the "bare" domain

So that's why the server response said it moved to https://www.bryancolvin.com/bender. The step you did not show is that if you try that with curl, you get another 301: your content is not at /bender, but at /bender/. The trailing slash indicates the content is a directory, not a file. In a plain web site there is an expected index.html in a directory and that is what you get when you ask for the directory itself. If you're returning JSON, then all bets are off -- it depends on how you do it.

With -L, curl followed both redirects for you. Anyway, try

    client.println("GET /bender/ HTTP/1.1");

BTW, the response headers say Content-Type: text/html, which is wrong. If you're consuming the JSON in your sketch by manually performing HTTP, then it doesn't matter. But if you intend for this data to be used by others, the type should be application/json

Perhaps more importantly, the server also responds with Transfer-Encoding: chunked, which complicates things if you're manually performing HTTP. The easy way to disable that is to switch your request to HTTP 1.0, which does not support it.

    client.println("GET /bender/ HTTP/1.0");

And then you also don't need to request Connection: close because that's what happens with 1.0 anyway.

1 Like

Thank you! This had the info I needed. I got it to spit out my JSON.

its an index.php file that write the JSON so I can server side mess with it.

Thanks a lot for pointing out more things for me to look out for. Helps me see some reading I need to dig into to understand servers better.

Thanks a million.