Http post with a long payload & read json response

Projects Programming
1

I know that everyone will ask for the full code to analyze, but I don't have it with me now and I need general guidance according to your experience. So please try to imagine the requirement and just try to help if you have an idea.

Problem:
I wan to use a 3rd party API. I need to send a huge payload (base64 encoded and put in a json format) and then receive json formatted response with a huse payload.
By huge ; I mean about 5KB to 30KB

Usually I use HttpClient library for my post requests. But in this case, I am not able to use it because HttpClient requires use of String and on Arduino long string is not possible. (If I am wrong and/or there is a way to use long String in HttpClient post, please let me know)

So I use WifiSecureClient and write the request in a while loop reading from an array using client.write() method.

This reaches the server (API) fine and the server responds with a usable json output.
But since I am using the WifiSecureClient, I get a raw response and I have to read and parse it.
So I skip the headers until I see json output. Then I try to get the json to a char* defined on PSRAM.
But the response is too long and I also see length definitions between the lines and it is not easy to parse.

Is there a library or method that I could use to get the response as a json output without cleaned from data length definitions ?

As the reponse is very long, I can't get it to a String variable.

What would you suggest ?

2

What microcontroller do you want to use?

3

I am using an Esp32s3

4

I've never used it though...how about this:

5

Start with the ArduinoJson library.
I don't see a reason why that library should have a problem with a large JSON.
Start with the ArduinoJson Assistant.
Let the HTTP Client stream into the ArduinoJso doc.

If you need an example, see my ESP Weather Monitor with OpenweatherMap example.

6

What do you want to do with this JSON?

That's probably Transfer-Encoding: chunked. The HTTPClient would decode that chunked encoding for you. Note that if the response headers include a Content-Length, then HTTPClient::getString will reserve that much space in the result String. The minimum-maximum length on ESP32 is just under 64KB. A near-empty sketch I compiled just now reported "leaving 309672 bytes for local variables". In addition to whatever else your sketch is actually doing, heap fragmentation could be an issue.

Do you actually get an invalid String from HTTPClient::getString? (Check as a boolean: if (s) evaluates as false)

IIRC, chunked and Content-Length can be mutually exclusive in practice. One way to disable chunked is to make the request with HTTP/1.0 instead of 1.1 -- 1.0 does not support chunked, but the server may reject such a request. The function is HTTPClient::useHTTP10

7

I know the ArduinoJson library and I use it on my other projects. In this project, the problem is not about parsing the json.
It is about posting long string payload.
I am not able to send the long string (json formatted) to the server fully.
is there a method that ArduinoJson has for posing data ?

8

here you describe that have a long response:

Can you pls clarify if you have problems with the client request or with the server response?

9

The problem starts on sending (post) , if you read my problem description; I am not able to use httpclient getstring because I need to use WifiSecureClient for posting.
If I can overcome that issue (how ?) of posting long string, then I can try httpclient getstring...

10

Both.

First problem is request. As I have a long string to post, HttpClient fails to send it.
As a solution to this "first" problem , I use WifiSecureClient.
So the first problem is resolved, but then starts the "second" problem: server response.

11

This is not huge, at least for an ESP32-S3.

I don't use the HTTPClient library very often, but with my Telegram library, for example, I've successfully tested sending pictures or documents of around 500–1000 KB without any issues. However, in these cases, I've directly used the WiFiClientSecure::write() method.

12

If the issue you're referring to involves the server's chunked responses, yes, it can be somewhat challenging to implement a parser. This is because the data typically needs sanitization to ensure you obtain correctly formatted JSON.

However, if your request is made using the HTTP/1.0 protocol and the server still supports it, the response won't be chunked as already stated from @kenb4

13

is there a way to use HttpClient with a long request payload (json) ?

14

so your poblem is that you need to use https to SEND and RECEIVE a large JSON?

15

I don't think that http/https will matter. The problem is sending a large json because String variable can't hold that much data.
Maybe if there was a way to send string in parts using httpclient, it would solve my problem.
Is there any method that would support this ?

16

The httpClient accepts following sendRequests

  int sendRequest(const char *type, String payload);
  int sendRequest(const char *type, uint8_t *payload = NULL, size_t size = 0);
  int sendRequest(const char *type, Stream *stream, size_t size = 0);

You are not limited to use only a String when you want to send much data.
I would try a Stream.

17

I don't know the sendRequest method. Usually I was using the HttpClient.POST(jsonpayload) method until I got this issue with long string.
Then I used the WifiClientSecure.print() method with chunked strings.
Now that HttpClient.sendRequest(type, stream, size) method might be helpful.
But how can I convert my long string to a stream ?

18

You don't have a long String in first place.
You should have a JSON object and you should stream the JSON object into the sendRequest.

19

ok I can put the string in a json object using arduinojson.
But then how shall I stream it ?

20

see the ArduinoJson Assistant. There should be an option to generate code with streaming output.