Handling text payload from HTTP GET on ESP 8266

I'm new to this. I have a board nodemcu-v3-esp8266-12e with a small screen.

I can display text (using u8g2, rather than Serial like in most examples).
I can connect to wifi.

What I want to do eventually is get weather forecasts and display to screen. I think this requires ESP8266HTTPClient, sending GET requests. I have trouble getting good response codes from wttr.in, so I am starting with asking for IP addresses using two websites you'll see below. These give response code 200.

My current problem is that the payload from getString prints out as a few (or often just one) nonsense characters. From "curl -v www.icanhazip.com", I expect the payload to be "Content-Type: text/plain", so I hoped that http.getString() would get me a String, which I think I have to convert to char[] to print.

I guess some questions are:
• Where are there examples I can follow? So many examples seem to assume internal or non-existent servers.
• How can I get a plain-text payload?
• If that is too easy, I'd welcome pointers on how to split up multiline payloads, assuming I get there eventually.
• Also, I've run out of ideas how to get any response from wttr.in -- I've tried "http://wttr.in" and "http://5.9.243.187" but they give code 301. I'm hoping that I won't need JSON or HTTPS.
Thanks!

Current code (which gives me response codes 200, and responses such as "ô" and "ä"):

#include <Arduino.h>
#include <Wire.h>
#include <U8g2lib.h>

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

const char *ssid = "MY_SSID";
const char *password = "MY_PASSWORD";
WiFiClient client;

/* This connects to the screen: */
U8G2_SSD1306_128X64_NONAME_F_SW_I2C
u8g2(U8G2_R0, 14, 12, U8X8_PIN_NONE);

#define LEN_STRINGO 30
char stringo[LEN_STRINGO];

#define FONT        u8g2_font_6x12_te
#define TEXT_Y_TOP  10
#define TEXT_Y_JUMP 13
#define TEXT_Y_MAX  63
int text_y = TEXT_Y_TOP;

#define NEWSCREEN(delayo) do{ \
  delay(delayo); \
  u8g2.clearBuffer(); \
  text_y = TEXT_Y_TOP; \
}while(0)

#define WRITE_LINE(...) do{ \
  if(text_y > TEXT_Y_MAX) NEWSCREEN(1000); \
  sprintf(stringo,__VA_ARGS__); \
  u8g2.drawStr(0,text_y,stringo); \
  u8g2.sendBuffer(); \
  text_y += TEXT_Y_JUMP; \
}while(0)

void
get_wifi()
{
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    WRITE_LINE("wifi search...");
  }
}

void
setup()
{
  char serverName[100];
  HTTPClient http;
  int httpCode;
  String payload;

  u8g2.begin();
  u8g2.setFont(FONT);
  NEWSCREEN(0);

  get_wifi();

  NEWSCREEN(0);
  sprintf(serverName,"http://httpbin.org/ip");
  http.begin(client,serverName); /* client is type WiFiClient, declared globally */
  httpCode = http.GET();
  WRITE_LINE("%d:%s", httpCode,serverName);

  payload = http.getString();
  WRITE_LINE("%s",payload);
  http.end();

  sprintf(serverName,"http://icanhazip.com");
  http.begin(client,serverName);
  httpCode = http.GET();
  WRITE_LINE("%d:%s", httpCode,serverName);
  WRITE_LINE("%s",http.getString());
  http.end();
}
void
loop()
{ // nothing here
}

Maybe ESP8266HTTPClient was a big dead-end for me.
Following How the read a website-content with ESP8266 ? - #4 by tolga121, I've used client.print(). Now I can get return code 1, but the output is just 3 blank lines. Here is setup(){ }, with the same headers and macros as before:

void
setup()
{
  char serverName[100];
  char request[200];
  int httpCode;
  String line;

  u8g2.begin();
  u8g2.setFont(FONT);
  NEWSCREEN(0);

  get_wifi();

  NEWSCREEN(0);
  sprintf(serverName,"icanhazip.com");
  httpCode = client.connect(serverName,80);
  WRITE_LINE("%d:%s", httpCode,serverName);
  sprintf(request,
      "GET / HTTP/1.1\r\n"
      "Host: %s\r\n"
      "User-Agent: ESP8266/NodeMCU 0.9\r\n"
      "Accept: */*\r\n",
      serverName);
  client.print(request);
  WRITE_LINE("have printed request");
  while(client.connected()){
    WRITE_LINE("connected");
    line = client.readStringUntil('\r');
    WRITE_LINE("%s",line);
  }

  WRITE_LINE("endo");
  delay(20000);
}

By the way, I've tried to follow the advice in arduino - Request cURL with ESP8266 - Stack Overflow to get the GET parameters (headers, maybe?) from "curl -v [url]".

is the correct start.
Follow the example BasicHttpClient.

this are the lines which will give you the received payload:

        // file found at server
        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          String payload = http.getString();
          Serial.println(payload);
        }

OK, thanks. I think that example is almost exactly what I have copied (from Arduino/libraries/ESP8266HTTPClient/examples/BasicHttpClient/BasicHttpClient.ino at master · esp8266/Arduino · GitHub). The biggest difference I can see in my code (apart from the screen library) is that I have not used WiFiMulti -- what is that? Do I need that, or is there some other reason why it is not working for me?

Thanks. I have now copied that example as exactly as I can, but still the text is nonsense. Could it be some kind of encoding issue with the String? The characters that are printed are nonsense but not random -- for example, ÿ is very frequent.

Here is my code. The httpCode values are 200, but the text is just a few nonsense characters.


#include <Arduino.h>
#include <Wire.h>  /* added */
#include <U8g2lib.h>  /* added */

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

#include <WiFiClient.h>

ESP8266WiFiMulti WiFiMulti;

/* This is the example from
   https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/
     examples/BasicHttpClient/BasicHttpClient.ino

 * The only differences are:
 *   screen library
 *   SSID and PASSWORD
 *   website contacted.
*/

/* This connects to the screen: */
U8G2_SSD1306_128X64_NONAME_F_SW_I2C
u8g2(U8G2_R0, 14, 12, U8X8_PIN_NONE);

#define LEN_STRINGO 30
char stringo[LEN_STRINGO];

#define FONT u8g2_font_6x12_te
#define TEXT_Y_TOP 10
#define TEXT_Y_JUMP 13
#define TEXT_Y_MAX 63
int text_y = TEXT_Y_TOP;

#define NEWSCREEN(delayo) \
  do { \
    delay(delayo); \
    u8g2.clearBuffer(); \
    text_y = TEXT_Y_TOP; \
  } while (0)

#define WRITE_LINE(...) \
  do { \
    if (text_y > TEXT_Y_MAX) NEWSCREEN(1000); \
    sprintf(stringo, __VA_ARGS__); \
    u8g2.drawStr(0, text_y, stringo); \
    u8g2.sendBuffer(); \
    text_y += TEXT_Y_JUMP; \
  } while (0)

void setup() {

  u8g2.begin();
  u8g2.setFont(FONT);
  NEWSCREEN(0);

  WRITE_LINE("setup...");

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(SSID, PASSWORD); /* #define earlier [deleted] */
}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    WiFiClient client;

    HTTPClient http;

    WRITE_LINE("[HTTP] begin...\n");
    if (http.begin(client, "http://www.httpbin.org/ip")) {  // HTTP


      WRITE_LINE("[HTTP] GET...\n");
      // start connection and send HTTP header
      int httpCode = http.GET();

      // httpCode will be negative on error
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        WRITE_LINE("%d:[HTTP] GET... code", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          String payload = http.getString();
          WRITE_LINE("%s",payload);
        }
      } else {
        WRITE_LINE("[HTTP] GET... failed, error:");
        WRITE_LINE("%s", http.errorToString(httpCode).c_str());
      }

      http.end();
    } else {
      WRITE_LINE("[HTTP] Unable to connect");
    }
  }

  delay(10000);
}

Google for it!

But it's just a library that lets you give a list of WiFi SSIDs & passwords and it automatically selects one that is available. A little like in the WiFi settings on your smartphone/laptop.

Screenshot_20240617-103432-066
LOL!

Yes, you're right -- I did Google for WiFiMulti right after I posted that message. I also tried a more literal copy of the example that @noiasca mentioned. I'm still stuck, though -- any thoughts?

I Googled for the significance of "ÿ" -- it's Unicode character U+00FF, so a -1 character can be cast to it. So I'm seeing -1 characters, meaning end-of-file or end-of-line or something.

Maybe I should try the "stream" example Arduino/libraries/ESP8266HTTPClient/examples/StreamHttpClient/StreamHttpClient.ino at master · esp8266/Arduino · GitHub -- I think this effectively gets the payload byte by byte (instead of http.getString). At least this will give me more idea of how many characters the website thinks it's sending. I'll try this later. Any other ideas?

use again the example from the example folder,
adopt your credentials and the target URI
Print it to Serial

if this is working - your display handling is not correct.

I could not follow your advice to use Serial -- I have no hardware that responds to those commands.

But you were correct that display handling was the problem. I found this out eventually by looking at each character of the payload individually. It turns out that %s does not work as a format for String -- I have to convert it to char array using a method for String. Also, \n stops subsequent characters from being displayed.

So now I can get icanhazip.com to tell me my IP address -- I recommend it as a simple website to test your http GET commands. I'm still having trouble with more complicated websites -- I'll bang my head against them for a while before I start a new post.

Thanks for your help!

Here is working code (for an Ideaspark ESP32 with mini OLED) for anyone interested:

#include <Arduino.h>
#include <Wire.h>  /* added */
#include <U8g2lib.h>  /* added */
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClient.h>
ESP8266WiFiMulti WiFiMulti;

/* This is the example from
   https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/
     examples/BasicHttpClient/BasicHttpClient.ino
 * The only differences are:
 *   screen library
 *   SSID and PASSWORD
 *   website contacted.
*/
#define SSID "XXXX"
#define PASSWORD "XXXX"

/* This connects to the screen: */
U8G2_SSD1306_128X64_NONAME_F_SW_I2C
u8g2(U8G2_R0, 14, 12, U8X8_PIN_NONE);

#define LEN_STRINGO 50
char stringo[LEN_STRINGO];

#define FONT u8g2_font_5x8_tf
#define TEXT_Y_TOP 6
#define TEXT_Y_JUMP 9
#define TEXT_Y_MAX 63
#define TEXT_X_MAX 25
int text_y = TEXT_Y_TOP;

#define NEWSCREEN(delayo) \
  do { \
    delay(delayo); \
    u8g2.clearBuffer(); \
    text_y = TEXT_Y_TOP; \
  } while (0)

#define WRITE_LINE(...) \
  do { \
    if (text_y > TEXT_Y_MAX) NEWSCREEN(3000); \
    sprintf(stringo, __VA_ARGS__); \
    u8g2.drawStr(0, text_y, stringo); \
    u8g2.sendBuffer(); \
    text_y += TEXT_Y_JUMP; \
  } while (0)

void setup() {
  u8g2.begin();
  u8g2.setFont(FONT);
  NEWSCREEN(0);

  WRITE_LINE("setup...");

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(SSID, PASSWORD); /* #define earlier [deleted] */
}

void loop() {
  int sizo,getto,g;
  char copystring[LEN_STRINGO];
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    WiFiClient client;
    HTTPClient http;

    WRITE_LINE("[HTTP] begin...\n");
    if (http.begin(client, "http://www.icanhazip.com/")) {  // HTTP

      WRITE_LINE("[HTTP] GET...\n");
      int httpCode = http.GET();

      if (httpCode > 0) {
        WRITE_LINE("%d:[HTTP] GET... code", httpCode);

        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          sizo=http.getSize();
          WRITE_LINE("%d:size %d:connected",sizo,(http.connected()?1:0));
          String payload = http.getString();
          payload.replace('\0','X');
          payload.replace('\n','N');
          payload.replace('\r','R');
          while(sizo>0){
            getto = sizo>TEXT_X_MAX-1 ? TEXT_X_MAX-1 : sizo;
            payload.toCharArray(copystring,getto+1);
            WRITE_LINE("%s",copystring);
            sizo-=getto;
            payload.remove(0,getto);
            delay(100);
          }
        }
      } else {
        WRITE_LINE("[HTTP] GET... failed, error:");
        WRITE_LINE("%s", http.errorToString(httpCode).c_str());
      }
      http.end();
    } else {
      WRITE_LINE("[HTTP] Unable to connect");
    }
  }
  delay(10000);
}

Of course you do. It is built in to Arduino.

Put Serial.begin(115200); in setup(), then use the Arduino IDE serial monitor, or any terminal program to read output from Serial.print().