API request and elaborate ESP32

Hello!
I have been trying to use an ESP32 to request the trash pickup schedule from my local waste management, but I keep getting an incomplete response.
The request I'm making is https://apps.hvcgroep.nl/rest/adressen/0479200000022388/afvalstromen
And the reply I get is

[
  {
    "id": 6,
    "parent_id": 0,
    "title": "plastic, blik & drinkpakken",
    "slug": null,
    "tags": null,
    "page_title": "plastic, blik & drinkpakken",
    "content": //a big bunch of useless information,
    "ophaaldatum": "2025-01-30"
  },
  ...
  //a bunch more similar elements in the array
  ...
]

and the way I do that is by

  HTTPClient http;

  http.begin(url);
  int httpResponseCode = http.GET();

  if (httpResponseCode > 0) {
    Serial.print("HTTP Response Code: ");
    Serial.println(httpResponseCode); 

    String payload = http.getString();

Pretty simple stuff so far, and I doubt the problem is in my code.

I think the problem lies in the fact that there is a whole bunch of useless information in the "content" node, we are talking 4000+ characters for every element in the array, which I don't know if I can filter out, that fill up my memory and I end up with an incomplete response.
When I then try to deserializeJson it throws up the error IncompleteInput and I can't do anything with that information.

Do you guys have any idea on how to solve that?
To be fair I am only really interested in getting the "title" and "ophaaldatum" of the next collection day, so I can elaborate it furhter.

The JSON you get back is more than 128 KB so you really can't load that in a string and parse it. you could explore ArduinoJson filters but may be going the low level way is easier ?

try this

you should see in the Serial monitor

of course, you'll need to adjust the WiFi network information with your specific details

const char* ssid = "xxxx";
const char* password = "xxx";

if all works fine you should see in the Serial monitor (at 115200 bauds)

Attempting to connect to SSID: Wokwi-GUEST
..Connected to Wokwi-GUEST

Starting connection to server...
Connected to server
FILTERED RESPONSE BODY
ID [6]	title [plastic, blik & drinkpakken]	Ophaaldatum: [2025-01-30]
ID [55]	title [Plastic gemeente Zaanstad]	Ophaaldatum: [null]
ID [5]	title [gft & etensresten]	Ophaaldatum: [2025-02-10]
ID [3]	title [papier en karton]	Ophaaldatum: [2025-02-05]
ID [209]	title [Papier en karton gemeente Zaanstad]	Ophaaldatum: [null]
ID [10]	title [glas]	Ophaaldatum: [null]
ID [8]	title [textiel]	Ophaaldatum: [null]
ID [2]	title [restafval]	Ophaaldatum: [null]
ID [11]	title [grof afval]	Ophaaldatum: [null]
ID [22]	title [Grof afval - gemeente Zaanstad]	Ophaaldatum: [null]
ID [13]	title [kringloop]	Ophaaldatum: [null]
ID [153]	title [Kringloop gemeente Zaanstad]	Ophaaldatum: [null]
ID [9]	title [afvalbrengstation]	Ophaaldatum: [null]
ID [200]	title [afvalbrengstation gemeente Zaanstad]	Ophaaldatum: [null]
ID [14]	title [plaagdieren]	Ophaaldatum: [null]
ID [133]	title [Plaagdieren gemeente Zaanstad]	Ophaaldatum: [null]
ID [57]	title [apparaten]	Ophaaldatum: [null]
ID [75]	title [Apparaten gemeente Zaanstad]	Ophaaldatum: [null]
ID [58]	title [klein chemisch afval (kca)]	Ophaaldatum: [null]
ID [99]	title [Klein chemisch afval (kca) gemeente Zaanstad]	Ophaaldatum: [null]

the way it works is basically I issue an HTTPS request to the server using the URL you shared, I then get a response and skip the headers (skip content until I get a blank line) and then I parse the body looking for the ID, then the title then the Ophaaldatum and again until there is nothing left to parse.

You need to be careful as the Ophaaldatum can be null so I look for "Ophaaldatum": and if the next character is a double quote, then I know I have a value, otherwise it's null and I don't bother reading it

There is no error checking, assumes the connexion will work fine and the response is formatted properly.

I left as an exercise for you to actually store the information that currently goes to the Serial monitor

Let me know if that works for you — seems it works in wokwi.

click to see the code

/* ============================================
  code is placed under the MIT license
  Copyright (c) 2025 J-M-L
  For the Arduino Forum : https://forum.arduino.cc/u/j-m-l

  Permission is hereby granted, free of charge, to any person obtaining a copy
  of this software and associated documentation files (the "Software"), to deal
  in the Software without restriction, including without limitation the rights
  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  copies of the Software, and to permit persons to whom the Software is
  furnished to do so, subject to the following conditions:

  The above copyright notice and this permission notice shall be included in
  all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  THE SOFTWARE.
  ===============================================
*/


#include <WiFi.h>
#include <WiFiClientSecure.h>
const char* ssid = "Wokwi-GUEST";
const char* password = "";

#define HOST "apps.hvcgroep.nl"
#define URL "https://" HOST "/rest/adressen/0479200000022388/afvalstromen"

const char* test_root_ca = \
                           "-----BEGIN CERTIFICATE-----\n" \
                           "MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB\n" \
                           "iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
                           "cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n" \
                           "BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw\n" \
                           "MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV\n" \
                           "BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU\n" \
                           "aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy\n" \
                           "dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK\n" \
                           "AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B\n" \
                           "3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY\n" \
                           "tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/\n" \
                           "Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2\n" \
                           "VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT\n" \
                           "79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6\n" \
                           "c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT\n" \
                           "Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l\n" \
                           "c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee\n" \
                           "UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE\n" \
                           "Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd\n" \
                           "BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G\n" \
                           "A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF\n" \
                           "Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO\n" \
                           "VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3\n" \
                           "ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs\n" \
                           "8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR\n" \
                           "iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze\n" \
                           "Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ\n" \
                           "XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/\n" \
                           "qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB\n" \
                           "VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB\n" \
                           "L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG\n" \
                           "jjxDah2nGN59PRbxYvnKkKj9\n" \
                           "-----END CERTIFICATE-----\n";



WiFiClientSecure client;

void setup() {
  Serial.begin(115200);
  while (!Serial);

  // attempt to connect to Wifi network:
  Serial.print("Attempting to connect to SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(500);
  }
  Serial.print("Connected to "); Serial.println(ssid);

  client.setCACert(test_root_ca);
  Serial.println("\nStarting connection to server...");

  if (!client.connect(HOST, 443)) {
    Serial.println("Connection failed!");
  } else {
    Serial.println("Connected to server");

    client.println("GET " URL " HTTP/1.0");
    client.println("Host: " HOST);
    client.println("User-Agent: ESP32");
    client.println("Content-Type: application/json; charset=utf-8");
    client.println("Connection: close");
    client.println();

    char c;
    bool currentLineIsEmpty = false;

    // skip the headers
    while (client.connected()) {
      if (client.available()) {
        c = client.read();
        if (c != '\n') {
          if (c != '\r') currentLineIsEmpty = false;
        } else if (currentLineIsEmpty) break;
        else currentLineIsEmpty = true;
      }
    }

    Serial.println("FILTERED RESPONSE BODY");
    while (client.connected()) {
      // look for "title"
      char c;


      if (client.find("\"id\":")) {
        Serial.print("ID [");
        while (client.available()) {
          c = client.read();
          if (c == ',') break;
          else Serial.write(c);
        }
        Serial.print("]\t");
      } // end find ID

      if (client.find("\"title\":\"")) {
        Serial.print("title [");
        while (client.available()) {
          c = client.read();
          if (c == '"') break;
          else Serial.write(c);
        }
        Serial.print("]\t");
      } // end find title

      // Now look for "ophaaldatum", value can be null and not a string starting with a double quote, so check for it
      if (client.find("\"ophaaldatum\":")) {
        Serial.print("Ophaaldatum: [");
        if (client.peek() == '\"') { // we have a string
          client.read(); // skip the "
          while (client.available()) {
            c = client.read();
            if (c == '"') break;
            else Serial.write(c);
          }
          Serial.println("]");
        } else { // it's null
          Serial.println("null]");
        }
      } // end find ophaaldatum
    }
  }
}

void loop() {}

Of course, if their REST API would allow you to define exactly what fields to extract it would be MUCH simpler...

out of curiousity I clicked on the link
oops!

looks like a lot of ehm - - - ... --- trash ?-Data?

it's real data

The JSON structure looks like this

you have a few short information fields and then the Content is some HTML and icon_data is a svg graphic encoded in base64. You can paste that right into your browser and you'll see what the icon look like

Some on line services offer a filtering capability to only get a few fields rather than the full JSON. if that's possible with https://apps.hvcgroep.nl/rest then that's what OP should use, non need to transfer tons of information you don't need.

Then entire string (small 's') could be read and stored in ESP32 PSRAM.

But, I've never used the ArduinoJson library. So, I'd need to look into whether it allows you to specify where its objects are created and processed (parsed). If it allows specifying that via point, then it could work.

Rather inefficient when such a small amount of data is need from such a large data structure. But, how often do you really need to download and parse the trash schedule? My township's schedule hasn't changed for the past ten years!

OK, it took a little research (and some code borrowed from @J-M-L), but here's a solution using PSRAM and the ArduinoJson library:

#include "Arduino.h"
#include <ArduinoJson.h>
#include <WiFi.h>
#include <NetworkClientSecure.h>

template<typename T>
void printItem(JsonDocument &document, const char *field) {
  JsonVariantConst v { document[field] };
  Serial.printf("[%s]: ", field);
  if (v.isNull()) {
    Serial.println("Not Found");
  } else {
    const T item { v };
    Serial.println(item);
  }
}

ARDUINOJSON_BEGIN_PUBLIC_NAMESPACE
class PsramAllocator: public Allocator {
  public:
    void *allocate(size_t size) override {
      return ps_malloc(size);
    }

    void deallocate(void *ptr) override {
      free(ptr);
    }

    void* reallocate(void *ptr, size_t new_size) override {
      return ps_realloc(ptr, new_size);
    }

    static Allocator* instance() {
      static PsramAllocator allocator;
      return &allocator;
    }

  private:
    PsramAllocator() = default;
    ~PsramAllocator() = default;
};
ARDUINOJSON_END_PUBLIC_NAMESPACE

struct CustomReader {
    CustomReader(NetworkClientSecure &client) : theClient(client) {
    }

    int read() {
      while (theClient.connected()) {
        while (theClient.available() == 0) {
          delay(1);
        }
        char c { static_cast<char>(theClient.read()) };
        if ((c == '[') || (c == ']')) {
          continue;
        }
        return c;
      }
      return -1;
    }

    // Not Yet Implemented
    size_t readBytes(char *buffer, size_t length) {
      return -1;
    }

  private:
    NetworkClientSecure &theClient;
};

extern const char *test_root_ca;

#define HOST "apps.hvcgroep.nl"
#define URL "https://" HOST "/rest/adressen/0479200000022388/afvalstromen"

void setup() {
  Serial.begin(115200);
  delay(1000);

  NetworkClientSecure esp32WiFiClient;
  CustomReader reader(esp32WiFiClient);
  const char ssid[] { "Hotspot" };
  const char password[] { "xxxxx" };
  uint8_t retryTest { 0 };
  JsonDocument doc(PsramAllocator::instance());

  while (WiFi.status() != WL_CONNECTED) {
    if (retryTest == 0) {
      WiFi.disconnect();
      WiFi.begin(ssid, password);
      retryTest = 5;
    } else {
      log_i("Waiting for WiFi Connection");
      retryTest--;
      delay(1000);
    }
  }
  Serial.println();
  Serial.print("WiFi connected. IP Address = ");
  Serial.println(WiFi.localIP());

  esp32WiFiClient.setCACert(test_root_ca);

  if (!esp32WiFiClient.connect(HOST, 443)) {
    Serial.println("Connection failed!");
  } else {
    Serial.println("Connected to server");

    esp32WiFiClient.println("GET " URL " HTTP/1.0");
    esp32WiFiClient.println("Host: " HOST);
    esp32WiFiClient.println("User-Agent: ESP32");
    esp32WiFiClient.println("Content-Type: application/json; charset=utf-8");
    esp32WiFiClient.println("Connection: close");
    esp32WiFiClient.println();

    // skip the headers
    char c;
    bool currentLineIsEmpty { false };
    while (esp32WiFiClient.connected()) {
      if (esp32WiFiClient.available()) {
        c = esp32WiFiClient.read();
        if (c != '\n') {
          if (c != '\r')
            currentLineIsEmpty = false;
        } else if (currentLineIsEmpty)
          break;
        else
          currentLineIsEmpty = true;
      }
    }

    DeserializationError error { deserializeJson(doc, reader) };
    if (error) {
      Serial.printf("deserializeJson() failed: %s\n", reinterpret_cast<const char*>(error.f_str()));
    } else {
      Serial.println("Parsing Succeeded");
    }

    printItem<unsigned int>(doc, "id");
    printItem<const char*>(doc, "title");
    printItem<const char*>(doc, "ophaaldatum");
  }
}

void loop() {

}

const char *test_root_ca =
  "-----BEGIN CERTIFICATE-----\n"
  "MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB\n"
  "iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n"
  "cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n"
  "BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw\n"
  "MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV\n"
  "BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU\n"
  "aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy\n"
  "dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK\n"
  "AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B\n"
  "3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY\n"
  "tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/\n"
  "Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2\n"
  "VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT\n"
  "79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6\n"
  "c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT\n"
  "Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l\n"
  "c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee\n"
  "UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE\n"
  "Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd\n"
  "BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G\n"
  "A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF\n"
  "Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO\n"
  "VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3\n"
  "ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs\n"
  "8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR\n"
  "iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze\n"
  "Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ\n"
  "XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/\n"
  "qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB\n"
  "VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB\n"
  "L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG\n"
  "jjxDah2nGN59PRbxYvnKkKj9\n"
  "-----END CERTIFICATE-----\n";

OUTPUT:

WiFi connected. IP Address = 192.168.137.192
Connected to server
Parsing Succeeded
[id]: 6
[title]: plastic, blik & drinkpakken
[ophaaldatum]: 2025-01-30

interesting thx for sharing

It took me a little time staring at your Skip Header code to understand how it works.

    // skip the headers
    while (client.connected()) {
      if (client.available()) {
        c = client.read();
        if (c != '\n') {
          if (c != '\r') currentLineIsEmpty = false;
        } else if (currentLineIsEmpty) break;
        else currentLineIsEmpty = true;
      }
    }

Very clever. It does indeed correctly trigger on the sequence: \r\n\r\n. But, if I'm tracing it through correctly, it looks like it will also trigger on \n\r\n. Is that a concern.

indeed

In theory it's not an issue as HTTP mandates the use of \r\n (carriage return + line feed) as the line-ending sequence for headers according to RFC 9112 , followed by an empty CRLF line

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.