Simple http get request guidance please

I would like to pull data off a neurio power meter in my electrical box. Pinging the device at the local ip address: 192.168.0.166/current-sample in a web browser reliably returns a string from which I could extract the info I need (those not easily). I have been trying to use the arduinohttpclient library to do so but I'm missing something. I have explored the: SimpleHTTPExample.ino and SimpleGET.ino without success. I do seem to connect to the KHostname[] = "192.168.0.166" - as I get an "OK" rather than a "connection failed". I have tried countless interactions of the kPath[] = "/current-sample"; such as:
kPath[] = "/current-sample/";
kPath[]= "192.168.0.166/current-sample";
kPath[]= "192.168.0.166/current-sample/";
kPath[]= "http:/192.168.0.166/current-sample";
kPath[]= "/";

I get an error response code of -3 which the documentation does not define that I can see.

Here is my output:

Attempting to connect to WPA SSID: X%_-^(O)^-_%X

connected

startedRequest ok

Getting response failed: -3

I feel I'm close to making this work but missing something basic as my understanding of http business is minimal. Appreciate any suggestions.

#include <ArduinoHttpClient.h>
#include <WiFiNINA.h>

#define ssid "X%_-^(O)^-_%X"
#define pass ""

const char kHostname[] = "192.168.0.166";

const char kPath[] = "/current-sample";
const int kNetworkTimeout = 30*1000;
const int kNetworkDelay = 1000;

WiFiClient c;
HttpClient http(c, kHostname);

void setup()
{
  //Initialize serial and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // attempt to connect to WiFi network:
  Serial.print("Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  while (WiFi.begin(ssid, pass) != WL_CONNECTED) {
    // unsuccessful, retry in 4 seconds

  }

  Serial.println("connected");
}

void loop()
{
  int err =0;
  
  err = http.get(kPath);
  if (err == 0)
  {
    Serial.println("startedRequest ok");

    err = http.responseStatusCode();
    if (err >= 0)
    {
      Serial.print("Got status code: ");
      Serial.println(err);

      // Usually you'd check that the response code is 200 or a
      // similar "success" code (200-299) before carrying on,
      // but we'll print out whatever response we get

      // If you are interesting in the response headers, you
      // can read them here:
      //while(http.headerAvailable())
      //{
      //  String headerName = http.readHeaderName();
      //  String headerValue = http.readHeaderValue();
      //}

      int bodyLen = http.contentLength();
      Serial.print("Content length is: ");
      Serial.println(bodyLen);
      Serial.println();
      Serial.println("Body returned follows:");
    
      // Now we've got to the body, so we can print it out
      unsigned long timeoutStart = millis();
      char c;
      // Whilst we haven't timed out & haven't reached the end of the body
      while ( (http.connected() || http.available()) &&
             (!http.endOfBodyReached()) &&
             ((millis() - timeoutStart) < kNetworkTimeout) )
      {
          if (http.available())
          {
              c = http.read();
              // Print out this character
              Serial.print(c);
             
              // We read something, reset the timeout counter
              timeoutStart = millis();
          }
          else
          {
              // We haven't got any data, so let's pause to allow some to
              // arrive
              delay(kNetworkDelay);
          }
      }
    }
    else
    {    
      Serial.print("Getting response failed: ");
      Serial.println(err);
    }
  }
  else
  {
    Serial.print("Connect failed: ");
    Serial.println(err);
  }
  http.stop();

  // And just stop, now that we've tried a download
  while(1);
}

which type of Arduino do you have?
when you try in your browser, is it HTTP or HTTPS?

The path is correct.

The not-HTTP status codes -- not 200, 404, etc -- are documented at the top of HttpClient.h (Seems like that first comment is in the wrong place.)

static const int HTTP_SUCCESS =0;
// The end of the headers has been reached.  This consumes the '\n'
// Could not connect to the server
static const int HTTP_ERROR_CONNECTION_FAILED =-1;
// This call was made when the HttpClient class wasn't expecting it
// to be called.  Usually indicates your code is using the class
// incorrectly
static const int HTTP_ERROR_API =-2;
// Spent too long waiting for a reply
static const int HTTP_ERROR_TIMED_OUT =-3;
// The response from the server is invalid, is it definitely an HTTP
// server?
static const int HTTP_ERROR_INVALID_RESPONSE =-4;

You got HTTP_SUCCESS and then HTTP_ERROR_TIMED_OUT, which means the request was sent, but you did not receive the complete initial response status-line in time. The default timeout is

    // Number of milliseconds that we'll wait in total without receiving any
    // data before returning HTTP_ERROR_TIMED_OUT (during status code and header
    // processing)
    static const int kHttpResponseTimeout = 30*1000;

Did it take 30 seconds to get the "failed" message? You should follow the SimpleGet example instead. After calling responseStatusCode(), you can then just call responseBody() to get the whole thing instead of what you're following from SimpleHttpExample.

If you point SimpleGet to Google

char serverAddress[] = "google.com";  // server address
int port = 80;

does it work? Since you have just an IP address, you can try that instead; it will send a slightly different set of default request headers.

HttpClient client = HttpClient(wifi, IPAddress(192,168,0,166), port);  // commas instead of dots in address

using 33iot and the device access is not secure so: http://192.168.0.166/current-sample
and the response from the device is <1 s on the browser

With SimpleHTTPExample the error = -3 is exactly at 30s

With the SimpleGet.ino example using both client definitions listed below I get a different timeout error (= -2) at ~40s

char serverAddress[] = "192.168.0.166"; // server address

int port = 8080;

WiFiClient wifi;

//HttpClient client = HttpClient(wifi, serverAddress, port);

HttpClient client = HttpClient(wifi, IPAddress(192,168,0,166), port);

with SimpleGet - both client definitions:

SSID: X%_-^(O)^-_%X

IP Address: 192.168.0.172

making GET request

Status code: -2

Response:

Wait five seconds

and I should add that in the SimpleGet example my get is:

client.get("/current-sample");

and it works with using google as a host

correction:
The SimpleHTTPExample works with Google as host. I cannot make the SimpleGet work at all with either:

char serverAddress[] = "www.google.com";  // server address
//char serverAddress[] = "8.8.8.8";  // server address

The default port for HTTP is 80. It's also the default value for the third port argument, so you can just omit it

HttpClient client = HttpClient(wifi, IPAddress(192,168,0,166));

(Because 80 is the default, 8080 is a common "custom port".)

(8.8.8.8 is Google's DNS service, not their HTTP web site.)

I had previously tried with port = 80, and now with port removed from the Get call without any change. If I use 172.217.12.110 as an IP address for google I at least get a response - that its moved to www.google.com. If I use a domain name such as www.google.com or any other domain it times out. It appears the syntax requires an IP address, not name. But my local IP address does not return a response.

I do not have to enter any credentials to browse the /current sample. Something is different between the request via the browser (chrome) and that here. Appreciate further advise.

#include <ArduinoHttpClient.h>
#include <WiFiNINA.h>

#define ssid "X%_-^(O)^-_%X"
#define pass ""

//char serverAddress[] = "www.google.com";  // server address
char serverAddress[] = "192.168.0.166";  // server address
int port = 8080;

WiFiClient wifi;
HttpClient client = HttpClient(wifi, serverAddress);
int status = WL_IDLE_STATUS;

void setup() {
  Serial.begin(9600);
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);

    // Connect to WPA/WPA2 network:
    status = WiFi.begin(ssid, pass);
  }

  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
}

void loop() {
  Serial.println("making GET request");
  //client.get("/current-sample");
  client.get("/current-sample");

  // read the status code and body of the response
  int statusCode = client.responseStatusCode();
  String response = client.responseBody();

  Serial.print("Status code: ");
  Serial.println(statusCode);
  Serial.print("Response: ");
  Serial.println(response);
  Serial.println("Wait five seconds");
  delay(5000);
}

"google.com" works for me, and I use it specifically because it returns that short message to use www. instead. Maybe that's worth pursuing if the following does not work.

Seems like if you have an IP address, it needs to be an IPAddress, not a string. In that case, ArduinoHttpClient doesn't send the Host header; but many web servers expect this. Others might return 400, but maybe the power meter hangs waiting for it. So try changing the HttpClient definition

IPAddress host_ip {192,168,0,166};
WiFiClient wifi;
HttpClient client = HttpClient(wifi, host_ip);

and then for the request, instead of a single get, it's

  Serial.println("making GET request");
  client.beginRequest();
  client.get("/current-sample");
  client.sendHeader("Host", host_ip.toString());
  client.endRequest();

In addition to Host, Chrome sends several others like Accept and then some more obscure ones. If necessary, you could duplicate it exactly if you examine the request in the browser's Network monitor and show the raw request headers. In that case, you should also call client.noDefaultRequestHeaders() during setup to suppress the default User-Agent header, so you can impersonate Chrome.

Also, if "google.com" is not working for you, you can use the IPAddress to connect and send Host: google.com and the web server wouldn't know the difference.

Is this: #define ssid "X%_-^(O)^-_%X" the SSID of your local wireless lan which you use to connect to the internet for general activities or is it the ssid of a network (access point) created by the Neurio power meter ?
I guess that since the password is blank, it is not your home wireless lan.

You've talked about Chrome. Did you have to explicitly connect to SSID "X%-^(O)^-%X" on the PC (or whatever device) running Chrome before being able to enter the URL 192.168.0.166/current-sample and can you see what IP address this PC obtained ? ( in Windows, if that is what it is, you have to search for network connections and look at the properties)

to the latter comment first:
"X%-^(O)^-%X" is my router SSID and I omitted the credentials from the post. Both my PC and the Neurio are connected to the same network. Its at 192.168.0.166 which seems to be a fixed IP and for what its worth my PC is at currently at: 192.168.0.248. I think I see what you were considering as the neurio does produce a wifi signal to connect to at initial setup. Its connection to my network seems robust as it responds whenever I browse its address.

as for the changes to the ArduinoHttpClient initiation using IPAdress rather than a string:
While I still don't get the response from the page it does changes the behavior. Rather than a 40s presumed timeout delay I get an immediate response back with the same -2 code. Nothing seems to change whether the .sendHeader is included or not. I did explore some of the other clients operations as shown below. I also inspected the elements of the page in chrome, both the main page and the .../current-sample page; but I'm not sure what that tells me frankly.

IPAddress host_ip {192,168,0,166};
WiFiClient wifi;
HttpClient client = HttpClient(wifi, host_ip);

------------------------------------

  Serial.println("making GET request");
  //client.get("/current-sample");
  //client.get("/current-sample");
  client.beginRequest();
  client.get("/current-sample");
  client.sendHeader("Host", host_ip.toString());
  Serial.print("Read: ");Serial.println(client.read());
Serial.print("ReadHeader: ");Serial.println(client.readHeader());
Serial.print("ReadHeaderName: ");Serial.println(client.readHeaderName());
Serial.print("ReadHeaderValue: ");Serial.println(client.readHeaderValue());
Serial.print("Status code: ");Serial.println(client.responseStatusCode());
Serial.print("Response: ");Serial.println(client.readString());
 
client.endRequest();

output:

making GET request
Read: -1
ReadHeader: 255
ReadHeaderName:
ReadHeaderValue:
Status code: -2
Response:

The code you posted with a bunch of read-whatever() before endRequest() will not work. You have to send -- and end -- the whole request before reading any part of the response. But I'm guessing you tried that first, and that didn't work either?

In Chrome, if the dev tools are open when you make a request, it will be listed under the Network tab, and selecting it, you can see the Headers, and sometimes there's a checkbox to see the Raw request


and starting with the single request-line GET whatever, followed by the headers (word-wrapped in the screenshot), and finished with a blank line: those are the bytes the web server sees.

You can remove the HTTP client from the situation and just perform HTTP manually with the WiFiClient.

Kenb4 - appreciate your patience with me!

I got the raw headers from chrome as you recommended and used the following wificlient syntax I found on another forum ... with success!

  if (wifi.connect(server, 80)) {
    Serial.println("connected to server");
    // Make a HTTP request:
    wifi.println("GET /current-sample HTTP/1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept-Encoding: gzip, deflate Accept-Language: en-US,en;q=0.9 Cache-Control: max-age=0 Connection: keep-alive Host: 192.168.0.166 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36");
    wifi.println("Host: 192.168.0.166");
    wifi.println("Connection: close");
    wifi.println();
    Serial.println(wifi);

And it returns:

HTTP/1.0 200 OK

Content-Type: application/json

Expires: Mon, 26 Jul 1997 05:00:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, max-age=0, post-check=0, pre-check=0

Pragma: no-cache

{"sensorId":"0x0000C47F510574D0","timestamp":"2024-11-18T01:42:08Z","channels":[{"type":"PHASE_A_CONSUMPTION","ch":1,"eImp_Ws":180540671769,"eExp_Ws":71116866739,"p_W":568,"q_VAR":-76,"v_V":122.231},{"type":"PHASE_B_CONSUMPTION","ch":2,"eImp_Ws":136611101412,"eExp_Ws":69930312707,"p_W":611,"q_VAR":44,"v_V":121.696},{"type":"NET","ch":3,"eImp_Ws":231078781704,"eExp_Ws":161328053714,"p_W":1256,"q_VAR":-169,"v_V":121.963},{"type":"GENERATION","ch":4,"eImp_Ws":199868688720,"eExp_Ws":93514821583,"p_W":-77,"q_VAR":136,"v_V":121.964},{"type":"CONSUMPTION","ch":5,"eImp_Ws":311532177825,"eExp_Ws":135336554295,"p_W":1179,"q_VAR":-33,"v_V":121.963}],"cts":[{"ct":1,"p_W":638,"q_VAR":-124,"v_V":122.234,"i_A":5.548},{"ct":2,"p_W":618,"q_VAR":-45,"v_V":121.692,"i_A":5.218},{"ct":3,"p_W":-7,"q_VAR":88,"v_V":121.701,"i_A":0.737},{"ct":4,"p_W":-70,"q_VAR":48,"v_V":122.228,"i_A":0.747}]}

disconnecting from server.

Would have never gotten here with the help - so thank you!

I understand there are some libraries for parsing the json response?

Didn't notice until I quoted this line that not only do you have Accept, but several other headers like User-Agent all in the same line, along with the GET on the request-line. So that should mean they don't actually matter. You can make your code smaller by trying just

    wifi.println("GET /current-sample HTTP/1.1");
    wifi.println("Host: 192.168.0.166");
    wifi.println("Connection: close");

If that actually doesn't work, then try putting each header in its own println statement, so your code reads better. And if that doesn't work, then you can revert and know that the server is busted and fickle.

The usual way to use ArduinoJson is with HTTPClient: the latter will skip the response headers so the former will parse the JSON body. The info there is still relevant if you're manually performing HTTP; turns out to be fairly simple with a handy Stream function, since the class hierarchy is WiFiClient > Client > Stream > Print

#include <WiFi.h>
#include <ArduinoJson.h>
#include "arduino_secrets.h"

WiFiClient wifi;

void setup() {
  Serial.begin(115200);

  WiFi.begin(SECRET_SSID, SECRET_PASS);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("\nWiFi connected.");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  static int countdown = 3;
  if (countdown && wifi.connect("mocktarget.apigee.net", 80)) {
    Serial.print("countdown: ");
    Serial.println(countdown--);
    wifi.println("GET /json HTTP/1.0"); // 1.0 to disable chunked encoding, for reading JSON
    wifi.println();                     // Also don't need Host or Connection: close
    if (wifi.find("\r\n\r\n")) {  // blank line between headers and body
      Serial.println("found end of response headers");
      StaticJsonDocument<100> j;
      DeserializationError error = deserializeJson(j, wifi);
      if (error) {
        Serial.print("deserializeJson() failed: ");
        Serial.println(error.f_str());
      } else {
        const char *city = j["city"];
        if (city) {
          Serial.print("city: ");
          Serial.println(city);
        } else {
          Serial.println("invalid JSON payload");
        }
      }
    } else {
      Serial.println("invalid HTTP response");
    }
    if (wifi.connected()) {  // with HTTP/1.0, server should close connection
      wifi.stop();
    } else {
      Serial.println("server disconnected");
    }
  }
  Serial.println("wait five seconds");
  delay(5000);
}

I think I'd give that a different name to avoid confusion as there's already an object named WiFi.

Fair point. A lot of examples use client. But then you can often have both WiFiClient and HTTPClient.

At least the WiFi is usually used sparingly in setup, in code that has probably been blindly pasted thousands of times and tweaked slightly. Few actually use it actively.

correct ... all it took was addition of the HTTP/1.1

I too came to the conclusion that naming the object wifi was wholly confusing and settled on the amazingly novel choice wfclient. And given that my string is entirely static except for the data, a few wfclient.readStringUntil(); has allowed me to pull off the power info without learning too much about json parsing at least for now.

And of course now that I'm where I wanted to be, the Neurio app has been behaving and receiving steady data from the Generac API for the last few days. But I think I have too much momentum not to put together at least a rudimentary dashboard with this data and improve on it later if the app goes south again.

Cheers!

I was curious, so I wrote a simple hexdumping server to see exactly what was being sent by the ArduinoHttpClient. It prints a blank line between every run of available bytes received.

02:17:44.596 -> 47 45 54                                          |GET|
02:17:44.793 -> 
02:17:44.793 ->          20 2F 63 75 72  72 65 6E 74 2D 73 61 6D  |    /current-sam|
02:17:44.826 -> 70 6C 65 20 48 54 54 50  2F 31 2E 31 0D 0A 55 73  |ple HTTP/1.1..Us|
02:17:44.826 -> 65 72 2D 41 67 65 6E 74  3A 20 41 72 64 75 69 6E  |er-Agent: Arduin|
02:17:44.826 -> 6F 2F 32 2E 32 2E 30 0D  0A 43 6F 6E 6E 65 63 74  |o/2.2.0..Connect|
02:17:44.826 -> 69 6F 6E 3A 20 63 6C 6F  73 65 0D 0A 48 6F 73 74  |ion: close..Host|
02:17:44.826 -> 3A 20 31 39 32 2E 31 36  38 2E 31 2E 31 39 0D 0A  |: 192.168.1.19..|
02:17:44.826 -> 0D 0A                                             |..|

There's a consistent delay, sometimes 200ms, between the verb GET and the rest of the request-line and headers. That's odd because the code is

    // Send the HTTP command, i.e. "GET /somepath/ HTTP/1.0"
    iClient->print(aHttpMethod);
    iClient->print(" ");

    iClient->print(aURLPath);
    iClient->println(" HTTP/1.1");

This is on ESP32. Switching to the included HTTPClient with

  http.useHTTP10(true);
  http.begin(wifi, host_ip.toString(), 80, "/current-sample");
  http.addHeader("Host", host_ip.toString());
  http.GET();

the request is received "contiguously" (maybe some buffer in the pipeline is 64 bytes?)

02:22:35.742 -> 47 45 54 20 2F 63 75 72  72 65 6E 74 2D 73 61 6D  |GET /current-sam|
02:22:35.742 -> 70 6C 65 20 48 54 54 50  2F 31 2E 30 0D 0A 48 6F  |ple HTTP/1.0..Ho|
02:22:35.742 -> 73 74 3A 20 31 39 32 2E  31 36 38 2E 31 2E 31 39  |st: 192.168.1.19|
02:22:35.742 -> 0D 0A 55 73 65 72 2D 41  67 65 6E 74 3A 20 45 53  |..User-Agent: ES|
02:22:35.775 -> 50 33 32 48 54 54 50 43  6C 69 65 6E 74 0D 0A 43  |P32HTTPClient..C|
02:22:35.775 -> 6F 6E 6E 65 63 74 69 6F  6E 3A 20 63 6C 6F 73 65  |onnection: close|
02:22:35.775 -> 0D 0A 0D 0A                                       |....|

If the web server is sensitive to that gap in the initial request-line, that would explain why ArduinoHttpClient does not work.

Google request won't return "HTTP/1.1 200 OK" on a port 80 request.
It will return "HTTP/1.1 302 https://www.google.com"
That is a redirect to https port 443.

How and when Google upgrades to https is a bit more complicated. A lot of HTTP done on Arduino is akin to

$ telnet google.com 80
Trying 2607:f8b0:4023:1000::64...
Connected to google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: google.com

HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/

The various web client classes add a thin layer on top of that.

This topic's issue is a web server on a local power meter that seems picky about the HTTP requests it receives.

On this forum it's not unusual to see code with a one-and-done while (client.available()) loop, assuming that a request or response will be sent in a single shot. That is not always the case (as demonstrated).