Solar inverter charge controller HTTP GET issues

Other Hardware Home Automation
1

Hi all,
Firstly thank you for your time reading this & especially if you can see the error of my ways.
I've still got a lot to learn!
I'm using an ESP8266 to send a HTTP GET request to api.openweathermap.org which sometimes works and sometimes is being cut off before fully populating the JSON document leading to "deserializeJson() failed: IncompleteInput" errors.

I've given the WiFiClient a large timeout window & the JSON documents (one response & one filter) overly large buffers for the ESP's diddy RAM allowance but I still cannot get it functioning reliably.

Please excuse the rough layout & lack of loop along with zero functions built out as I wanted to get the bare bones behaving before progressing further.

Here's the code so far:

#include <ArduinoJson.h>
#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include "config.h"

const char* weatherHost = "api.openweathermap.org";
const int weatherPort = 80;
const int EntryCount = 8; //Aiming for 8 Entries
String weatherResource = "/data/2.5/forecast?id=" + weatherCityID + "&cnt=" + EntryCount + "&appid=" + weatherAPIKey;
int cloudCoverage;
String weatherData;

// Enable/Disable debugging client response print out
bool debug = false;

// Enable/Disable automatic reset from "deserializeJson() failed: IncompleteInput" error
bool autorestart = true;

void setup() {

  // Initialize Serial port
  Serial.begin(9600);
  Serial.println();

  // Initialize Wifi 
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(2500);
    //Serial.println("Connecting to WiFi...");
  }
  Serial.println();
  Serial.println("Connected to WiFi!");
  WiFiClient client;
  client.setTimeout(20000);  // 20 seconds - Massive overkill timeout

  //Connect to Host API
  if (!client.connect(weatherHost, weatherPort)) {
    Serial.println("Failed to connect to weather API");
    return;
  }
  Serial.println();
  Serial.println(F("Connected to API!"));
 
  // Send HTTP request
  String request = "GET " + weatherResource + " HTTP/1.1\r\n" +
                   "Host: " + weatherHost + "\r\n" +
                   "Connection: close\r\n\r\n";
  client.print(request);
  //Serial.print(request);
  //Serial. println();
  if (client.println(request) == 0) {
    Serial.println(F("Failed to send request"));
    client.stop();
    return;
  }
  
  // Print Response (Debugging use only as it Empties the client stream)
  String response = "";
  while (client.connected(), debug == true) {
    if (client.available()) {
      response = client.readString();
      Serial.println(response);
    }
  }
  
  // Check HTTP status
  char status[32] = {0};
  client.readBytesUntil('\r', status, sizeof(status));
  if (strcmp(status, "HTTP/1.1 200 OK") != 0) {
    Serial.print(F("Unexpected response: "));
    Serial.println(status);
    client.stop();
    return;
  }

  // Skip HTTP headers
  char endOfHeaders[] = "\r\n\r\n";
  if (!client.find(endOfHeaders)) {
    Serial.println(F("Invalid response"));
    client.stop();
    return;
  }

  // Allocate the JSON filter
  StaticJsonDocument<800> filter;

  // The filter: it contains "true" for each value we want to keep
  filter["list"][0]["dt"] = true;
  filter["list"][0]["clouds"]["all"] = true;
  filter["list"][0]["dt_txt"] = true;
  filter["city"]["sunrise"] = true;
  filter["city"]["sunset"] = true;

  // Allocate the JSON document
  DynamicJsonDocument doc(8000);

  // Parse JSON object
  DeserializationError error = deserializeJson(doc, client, DeserializationOption::Filter(filter));
  if (error) {
    Serial.println();
    Serial.print(F("deserializeJson() failed: "));
    Serial.println(error.f_str());
    String ErrorReturn = error.f_str();
    Serial.println();
    delay(3000);
      if (ErrorReturn == "IncompleteInput", autorestart == true){
        ESP.restart();
      }
    client.stop();
    return;
  }

  /*
  // Print restructured & filtered JSON document
  Serial.println();
  serializeJsonPretty(doc, Serial);
  Serial.println();
  */

  //Filter out JSON formatting of "clouds" to leave integers
  int clouds0 = doc["list"][0]["clouds"]["all"];
  int clouds1 = doc["list"][1]["clouds"]["all"];
  int clouds2 = doc["list"][2]["clouds"]["all"];
  int clouds3 = doc["list"][3]["clouds"]["all"];
  int clouds4 = doc["list"][4]["clouds"]["all"];
  int clouds5 = doc["list"][5]["clouds"]["all"];
  int clouds6 = doc["list"][6]["clouds"]["all"];
  int clouds7 = doc["list"][7]["clouds"]["all"];;

  //Average the cloud cover out
  int clouds = ((clouds0 + clouds1 + clouds2 + clouds3 + clouds4 + clouds5 + clouds6 + clouds7) / EntryCount);

  // Print the result
  Serial.println();
  Serial.print("Cloud coverage: ");
  Serial.print(clouds);
  Serial.println("%");
  Serial.println();

  // Disconnect
  client.stop();  
}

void loop() {
  
}

PS - the ssid, password, weatherAPIKey & weatherCityID are housed within config.h

The serial monitor response is generally:


Connected to WiFi!

Connected to API!

deserializeJson() failed: IncompleteInput

Connected to WiFi!

Connected to API!

deserializeJson() failed: IncompleteInput

Connected to WiFi!

Connected to API!

Cloud coverage: 80%

If I enable my debug "mode" I can see the client response is getting cut off part way through (missing the end) most of the time & only occasionally getting the complete JSON document.

The end game goal is to have the daylight cloud cover averaged out for the next day and if it's above a threshold value to send a command to the Solar inverter via RS485 to charge off of the grid in in the early morning to make use of a off peak - low cost energy tariff if the solar panels aren't likely to receive enough flux juice to fill up the batteries by themselves.

...Some ways to go before these bits get added!

Have I made a rookie/n00b error or am I just hitting the wall of the ESP8266's capabilities and need an MCU with more RAM to juggle stuffs?

Many thanks again for any help & insights!

Sam

2

IMHO as "readString()" is a non-blocking function, it does not read the whole data but only what has beed already received (stored inside the buffer), and that's why you often get an uncomplete response.
Try "readStringUntil('\n')" instead, or use "client.get(request)" instead of "client.print(request);"

EDIT: why did you print twice the request?

  client.print(request);
  //Serial.print(request);
  //Serial. println();
  if (client.println(request) == 0) {
1 Like
3

Thank so much for your helpful reply!
I ended up trying a ESP32 instead as part of my fault finding/bug bashing escapades which no longer accepted the client.readString() function after switching from the ESP8266WiFi.h library to the WiFi.h one.

The code seems to now run without returning the deserializeJson() failed: IncompleteInput error.

Instead I am using:

String response = "";  // Declare "response" String variable and clear it
    while (client.available()) {
      char c = client.read();
      response += c;
    }

From your reply I'm assuming the client.read() function falls into the same category of a non-blocking function which as you rightly pointed out is not ideal.

Is there a way to make sure the whole page is loaded into the client stream buffer before running the client.available() function while loop or the deserializeJson() function?
Is it just a cast of sticking a delay() in between sending the request and using the response and hoping for the best or can I detect the end of a valid JSON document or webpage before trying to deserialise it without reading and thus emptying the buffer?
I would like to run the deserializeJson() function using the client stream instead of storing it as a string as the ArduinoJSON troubleshooter on their website states a stream is more efficient for it.

Well spotted regarding my double printing of the client request! Thank you!
I've knocked out the first instance and left the request that is in the "if" statement as an error check.

I tried to implement the client.get(request) method but now I am using the WiFi.h library it errors stating there is no valid member get in the WiFiClient class sadly.

Many thanks indeed for your time & helpful reply!

4

Good. This way you manually read byte per byte the response and collect everything into the buffer string.
The official ESP site shows this as an example of response reading and I think it could be a bit quicker:

while (client.connected() || client.available())
{
  if (client.available())
  {
    String line = client.readStringUntil('\n');
    Serial.println(line);
  }
}

Please note the "|| client.available()" additional condition: I think it could happen the client is not connected anymore but some data is still inside the library internal receive buffer.
On that example the response is read line-by-line, you can just add each line to your buffer:

String response = ""; 
while (client.connected() || client.available())
{
  if (client.available())
    Response += client.readStringUntil('\n');
}

I never tested it, but AFAIK either using the above code or using the method "client.get()" I cited before, instead of the "while available+read" should do the trick.

PS: the timeout you set is the connection timeout.

1 Like
5

Apologies for the late reply.

It seems to be behaving itself now especially with the tips you've given me.

Thank you very much for all your time, input & help in general; It's been invaluable!

6

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.