Stuck Parsing JSON Data when Key does not Exist

Hi,
I am probably missing something very simple here but I have been trying for a while now to figure this out.

I am scraping an API: https://api.met.no

to get the Moon rise & set times for the day.

The API call has to be made as: https://api.met.no/weatherapi/sunrise/2.0/.json?lat=51.00045&lon=-3.77076&date=2020-09-11&offset=+00:00
with lat & lon plus the date in format YYYY-MM-DD, which is fine.

For say ‘normal’ days this is not a problem where The Moon rises & then sets within the same day but sometimes it will happen that The Moon can rise on the previous day & set on the current day, or rise on the current day & set on the following day.

The API that I am scraping only shows the rise & set times if they fall on the current day, the day set in the API call URL.

I am using ArduinoJSON to parse the JSON data & collect the rise & set times but when either the ‘moonrise’ or ‘moonset’ key is not present it breaks my project, obviously as ArduinoJSON cannot parse what is not there.

To try to overcome this I am using the Json::containsKey() function to check the JSON data for the specific keys.

With testing i.e. printing to the Serial Monitor I can see that my checks using ‘containsKey()’ are functioning but for some reason the procedures that I have put in place if the Key is not found, are not working.

Basically if the Key checking does not find the ‘moonrise’ key in the JSON data from the API call, it calls another function which makes another call to the API but this time with the previous days date in the URL, as The Moon must have risen on the previous day.

Similarly, if the Key checking does not find the ‘moonset’ key in the JSON data, another function is called to call the API with the following days date in the URL.

Hope this is making sense so far :wink:

Again, via testing with printing to Serial Monitor, I am able to see that initially these new functions are called but it stops short of completing it’s task fully and does not make the relative API call to collect the new data.

Probably related but I also have an issue where neither the value for ‘moonrise’ or ‘moonset’ are being printed to the Serial Monitor via the code I have in my main loop, understandably it will not be able to print data that is not there but even when there are values something strange is happening.

I am suspecting it has something to do with the HTTP Clients & or the setting insecure coding but like I said, I just need a fresh pair of eyes to spot the glaring error, please.

Oh, I make the initial API call with the date: 2020.09.12 as this particular date only returns a ‘moonset’ key, the Moon rise was actually the day before at 23:08:44UTC, so my code does find that there is no ‘moonrise’ key & then attempts to make a call using the previous days date but that as far as it goes.

Here’s my pretty rough code but in full:

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <ArduinoJson.h>
#include <WiFiClientSecureBearSSL.h>
#include <WiFiClientSecure.h>

const char* wifissid = "***";  // your network SSID (name)
const char* wifipass = "***";   // your network password


//Moon
const char* moonrise;
const char* moonset;


void setup() {
  Serial.begin(115200);
  WiFi.begin(wifissid, wifipass);
  Serial.println();
  Serial.print("Connecting.");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println();
  Serial.print("Connected to: ");
  Serial.print(wifissid);
  Serial.println();
}

void loop() {
  ReadMoonData();
  Serial.println("Below are the values printed in the main loop");
  Serial.println(moonrise);
  Serial.println(moonset);
  delay(3600000);
}




void ReadMoonData()
{
  std::unique_ptr<BearSSL::WiFiClientSecure>clientM(new BearSSL::WiFiClientSecure);
  clientM->setInsecure();

  HTTPClient httpsMoon;  //Object of class HTTPClient

  Serial.print("[HTTPS] begin...\n");
  if (httpsMoon.begin(*clientM, "https://api.met.no/weatherapi/sunrise/2.0/.json?lat=51.00045&lon=-3.77076&date=2020-09-12&offset=+00:00"))
  {
    int httpCodeMoon = httpsMoon.GET();

    if (httpCodeMoon > 0) {
      if (httpCodeMoon == HTTP_CODE_OK || httpCodeMoon == HTTP_CODE_MOVED_PERMANENTLY) {

        const size_t capacity = JSON_ARRAY_SIZE(2) + JSON_OBJECT_SIZE(1) + 6 * JSON_OBJECT_SIZE(2) + 5 * JSON_OBJECT_SIZE(3) + 2 * JSON_OBJECT_SIZE(4) + 2 * JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12) + 1740;

        StaticJsonDocument<200> filterM;
        filterM["location"]["time"][0]["moonrise"]["time"] = true;
        filterM["location"]["time"][0]["moonset"]["time"] = true;

        DynamicJsonDocument docM(capacity);
        deserializeJson(docM, httpsMoon.getString(), DeserializationOption::Filter(filterM));

        serializeJsonPretty(docM, Serial);

        Serial.print(httpsMoon.getString());

        JsonObject LOCATION = docM["location"];

        JsonObject TIME = LOCATION["time"][0];
        //moonrise = TIME["moonrise"]["time"];
        //moonset = TIME["moonset"]["time"];
        bool hasMR = TIME.containsKey("moonrise");
        if (!hasMR) {
          Serial.println();
          Serial.println("No Moonrise today");
          dayBefore();
        }
        else
        {
          moonrise = TIME["moonrise"]["time"];
          Serial.println();
          Serial.println("Moonrise today");
          Serial.println(moonrise);
        }
        bool hasMS = TIME.containsKey("moonset");
        if (!hasMS) {
          Serial.println();
          Serial.println("No Moonset today");
          dayAfter();
        }
        else
        {
          moonset = TIME["moonset"]["time"];
          Serial.println();
          Serial.println("Moonset today");
          Serial.println(moonset);
        }
        docM.clear();
      }
    }
    httpsMoon.end();

  }
}

void dayBefore()
{
  std::unique_ptr<BearSSL::WiFiClientSecure>clientMdB(new BearSSL::WiFiClientSecure);
  clientMdB->setInsecure();

  HTTPClient httpsMoondB;  //Object of class HTTPClient

  Serial.print("[HTTPS] begin...dayBefore\n");
  if (httpsMoondB.begin(*clientMdB, "https://api.met.no/weatherapi/sunrise/2.0/.json?lat=51.00045&lon=-3.77076&date=2020-09-11&offset=+00:00"))
  {
    int httpCodeMoondB = httpsMoondB.GET();

    if (httpCodeMoondB > 0) {
      if (httpCodeMoondB == HTTP_CODE_OK || httpCodeMoondB == HTTP_CODE_MOVED_PERMANENTLY) {

        const size_t capacity = JSON_ARRAY_SIZE(2) + JSON_OBJECT_SIZE(1) + 6 * JSON_OBJECT_SIZE(2) + 5 * JSON_OBJECT_SIZE(3) + 2 * JSON_OBJECT_SIZE(4) + 2 * JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12) + 1740;

        StaticJsonDocument<200> filterMdB;
        filterMdB["location"]["time"][0]["moonrise"]["time"] = true;

        DynamicJsonDocument docMdB(capacity);
        deserializeJson(docMdB, httpsMoondB.getString(), DeserializationOption::Filter(filterMdB));

        serializeJsonPretty(docMdB, Serial);

        Serial.print(httpsMoondB.getString());

        JsonObject LOCATION = docMdB["location"];

        JsonObject TIME = LOCATION["time"][0];
        moonrise = TIME["moonrise"]["time"];
        docMdB.clear();
      }
    }
    httpsMoondB.end();

  }
}

void dayAfter()
{
  std::unique_ptr<BearSSL::WiFiClientSecure>clientMdA(new BearSSL::WiFiClientSecure);
  clientMdA->setInsecure();

  HTTPClient httpsMoondA;  //Object of class HTTPClient

  Serial.print("[HTTPS] begin...dayAfter\n");
  if (httpsMoondA.begin(*clientMdA, "https://api.met.no/weatherapi/sunrise/2.0/.json?lat=51.00045&lon=-3.77076&date=2020-09-13&offset=+00:00"))
  {
    int httpCodeMoondA = httpsMoondA.GET();

    if (httpCodeMoondA > 0) {
      if (httpCodeMoondA == HTTP_CODE_OK || httpCodeMoondA == HTTP_CODE_MOVED_PERMANENTLY) {

        const size_t capacity = JSON_ARRAY_SIZE(2) + JSON_OBJECT_SIZE(1) + 6 * JSON_OBJECT_SIZE(2) + 5 * JSON_OBJECT_SIZE(3) + 2 * JSON_OBJECT_SIZE(4) + 2 * JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12) + 1740;

        StaticJsonDocument<200> filterMdA;
        filterMdA["location"]["time"][0]["moonset"]["time"] = true;

        DynamicJsonDocument docMdA(capacity);
        deserializeJson(docMdA, httpsMoondA.getString(), DeserializationOption::Filter(filterMdA));

        serializeJsonPretty(docMdA, Serial);

        Serial.print(httpsMoondA.getString());

        JsonObject LOCATION = docMdA["location"];

        JsonObject TIME = LOCATION["time"][0];
        moonset = TIME["moonset"]["time"];
        docMdA.clear();
      }
    }
    httpsMoondA.end();

  }

Thanks for looking.

Rather than making another call within the processing of the current call, you can refactor you code such that you make the call for a given date. If the result does not have moon rising, you call the same "fetch" function with the previous date. If the result does not have moon setting, you call the same "fetch" function with tomorrow.s date. This way, you complete the transaction completely before making another

blh64:
Rather than making another call within the processing of the current call, you can refactor you code such that you make the call for a given date. If the result does not have moon rising, you call the same "fetch" function with the previous date. If the result does not have moon setting, you call the same "fetch" function with tomorrow.s date. This way, you complete the transaction completely before making another

Ok, thanks for replying, I'll try the way you suggest & post back findings.