Pretty simple stuff so far, and I doubt the problem is in my code.
I think the problem lies in the fact that there is a whole bunch of useless information in the "content" node, we are talking 4000+ characters for every element in the array, which I don't know if I can filter out, that fill up my memory and I end up with an incomplete response.
When I then try to deserializeJson it throws up the error IncompleteInput and I can't do anything with that information.
Do you guys have any idea on how to solve that?
To be fair I am only really interested in getting the "title" and "ophaaldatum" of the next collection day, so I can elaborate it furhter.
The JSON you get back is more than 128 KB so you really can't load that in a string and parse it. you could explore ArduinoJson filters but may be going the low level way is easier ?
try this
you should see in the Serial monitor
of course, you'll need to adjust the WiFi network information with your specific details
if all works fine you should see in the Serial monitor (at 115200 bauds)
Attempting to connect to SSID: Wokwi-GUEST
..Connected to Wokwi-GUEST
Starting connection to server...
Connected to server
FILTERED RESPONSE BODY
ID [6] title [plastic, blik & drinkpakken] Ophaaldatum: [2025-01-30]
ID [55] title [Plastic gemeente Zaanstad] Ophaaldatum: [null]
ID [5] title [gft & etensresten] Ophaaldatum: [2025-02-10]
ID [3] title [papier en karton] Ophaaldatum: [2025-02-05]
ID [209] title [Papier en karton gemeente Zaanstad] Ophaaldatum: [null]
ID [10] title [glas] Ophaaldatum: [null]
ID [8] title [textiel] Ophaaldatum: [null]
ID [2] title [restafval] Ophaaldatum: [null]
ID [11] title [grof afval] Ophaaldatum: [null]
ID [22] title [Grof afval - gemeente Zaanstad] Ophaaldatum: [null]
ID [13] title [kringloop] Ophaaldatum: [null]
ID [153] title [Kringloop gemeente Zaanstad] Ophaaldatum: [null]
ID [9] title [afvalbrengstation] Ophaaldatum: [null]
ID [200] title [afvalbrengstation gemeente Zaanstad] Ophaaldatum: [null]
ID [14] title [plaagdieren] Ophaaldatum: [null]
ID [133] title [Plaagdieren gemeente Zaanstad] Ophaaldatum: [null]
ID [57] title [apparaten] Ophaaldatum: [null]
ID [75] title [Apparaten gemeente Zaanstad] Ophaaldatum: [null]
ID [58] title [klein chemisch afval (kca)] Ophaaldatum: [null]
ID [99] title [Klein chemisch afval (kca) gemeente Zaanstad] Ophaaldatum: [null]
the way it works is basically I issue an HTTPS request to the server using the URL you shared, I then get a response and skip the headers (skip content until I get a blank line) and then I parse the body looking for the ID, then the title then the Ophaaldatum and again until there is nothing left to parse.
You need to be careful as the Ophaaldatum can be null so I look for "Ophaaldatum": and if the next character is a double quote, then I know I have a value, otherwise it's null and I don't bother reading it
There is no error checking, assumes the connexion will work fine and the response is formatted properly.
I left as an exercise for you to actually store the information that currently goes to the Serial monitor
Let me know if that works for you — seems it works in wokwi.
click to see the code
/* ============================================
code is placed under the MIT license
Copyright (c) 2025 J-M-L
For the Arduino Forum : https://forum.arduino.cc/u/j-m-l
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
===============================================
*/
#include <WiFi.h>
#include <WiFiClientSecure.h>
const char* ssid = "Wokwi-GUEST";
const char* password = "";
#define HOST "apps.hvcgroep.nl"
#define URL "https://" HOST "/rest/adressen/0479200000022388/afvalstromen"
const char* test_root_ca = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB\n" \
"iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
"cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n" \
"BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw\n" \
"MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV\n" \
"BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU\n" \
"aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy\n" \
"dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK\n" \
"AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B\n" \
"3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY\n" \
"tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/\n" \
"Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2\n" \
"VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT\n" \
"79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6\n" \
"c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT\n" \
"Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l\n" \
"c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee\n" \
"UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE\n" \
"Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd\n" \
"BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G\n" \
"A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF\n" \
"Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO\n" \
"VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3\n" \
"ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs\n" \
"8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR\n" \
"iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze\n" \
"Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ\n" \
"XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/\n" \
"qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB\n" \
"VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB\n" \
"L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG\n" \
"jjxDah2nGN59PRbxYvnKkKj9\n" \
"-----END CERTIFICATE-----\n";
WiFiClientSecure client;
void setup() {
Serial.begin(115200);
while (!Serial);
// attempt to connect to Wifi network:
Serial.print("Attempting to connect to SSID: ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
Serial.print(".");
delay(500);
}
Serial.print("Connected to "); Serial.println(ssid);
client.setCACert(test_root_ca);
Serial.println("\nStarting connection to server...");
if (!client.connect(HOST, 443)) {
Serial.println("Connection failed!");
} else {
Serial.println("Connected to server");
client.println("GET " URL " HTTP/1.0");
client.println("Host: " HOST);
client.println("User-Agent: ESP32");
client.println("Content-Type: application/json; charset=utf-8");
client.println("Connection: close");
client.println();
char c;
bool currentLineIsEmpty = false;
// skip the headers
while (client.connected()) {
if (client.available()) {
c = client.read();
if (c != '\n') {
if (c != '\r') currentLineIsEmpty = false;
} else if (currentLineIsEmpty) break;
else currentLineIsEmpty = true;
}
}
Serial.println("FILTERED RESPONSE BODY");
while (client.connected()) {
// look for "title"
char c;
if (client.find("\"id\":")) {
Serial.print("ID [");
while (client.available()) {
c = client.read();
if (c == ',') break;
else Serial.write(c);
}
Serial.print("]\t");
} // end find ID
if (client.find("\"title\":\"")) {
Serial.print("title [");
while (client.available()) {
c = client.read();
if (c == '"') break;
else Serial.write(c);
}
Serial.print("]\t");
} // end find title
// Now look for "ophaaldatum", value can be null and not a string starting with a double quote, so check for it
if (client.find("\"ophaaldatum\":")) {
Serial.print("Ophaaldatum: [");
if (client.peek() == '\"') { // we have a string
client.read(); // skip the "
while (client.available()) {
c = client.read();
if (c == '"') break;
else Serial.write(c);
}
Serial.println("]");
} else { // it's null
Serial.println("null]");
}
} // end find ophaaldatum
}
}
}
void loop() {}
Of course, if their REST API would allow you to define exactly what fields to extract it would be MUCH simpler...
you have a few short information fields and then the Content is some HTML and icon_data is a svg graphic encoded in base64. You can paste that right into your browser and you'll see what the icon look like
Some on line services offer a filtering capability to only get a few fields rather than the full JSON. if that's possible with https://apps.hvcgroep.nl/rest then that's what OP should use, non need to transfer tons of information you don't need.
Then entire string (small 's') could be read and stored in ESP32 PSRAM.
But, I've never used the ArduinoJson library. So, I'd need to look into whether it allows you to specify where its objects are created and processed (parsed). If it allows specifying that via point, then it could work.
Rather inefficient when such a small amount of data is need from such a large data structure. But, how often do you really need to download and parse the trash schedule? My township's schedule hasn't changed for the past ten years!
It took me a little time staring at your Skip Header code to understand how it works.
// skip the headers
while (client.connected()) {
if (client.available()) {
c = client.read();
if (c != '\n') {
if (c != '\r') currentLineIsEmpty = false;
} else if (currentLineIsEmpty) break;
else currentLineIsEmpty = true;
}
}
Very clever. It does indeed correctly trigger on the sequence: \r\n\r\n. But, if I'm tracing it through correctly, it looks like it will also trigger on \n\r\n. Is that a concern.
In theory it's not an issue as HTTP mandates the use of \r\n (carriage return + line feed) as the line-ending sequence for headers according to RFC 9112 , followed by an empty CRLF line