I am sending http requests on my Giga r1 to Spotify api and it takes about 3 seconds, in which I can't do anything. I tried different libraries for multithreading but they don't work.
How can I made these requests not block main thread?
3 seconds to get the first byte in response? What kind of response is it? Do you use the body content?
If you perform HTTP manually, you can wait for and then pull the response as part of the main loop
#include <WiFi.h>
#include "arduino_secrets.h"
WiFiClient wifi;
bool hostConnected;
void setup() {
Serial.begin(115200);
WiFi.begin(SECRET_SSID, SECRET_PASS);
for (int i = 0; WiFi.status() != WL_CONNECTED; i++) {
Serial.print(i % 10 ? "." : "\n.");
delay(100);
}
Serial.println();
Serial.println(WiFi.localIP());
if (wifi.connect("www.google.com", 80)) {
hostConnected = true;
wifi.println("GET / HTTP/1.0"); // 1.0 means not having to request Connection: close, nor set the Host:
wifi.println("User-Agent: Arduino"); // without a UA, get 55KB of content instead of "normal" 18KB
wifi.println();
}
}
void checkWiFiIncoming() {
if (!hostConnected) {
return;
}
if (!wifi.connected()) {
hostConnected = false;
Serial.println("host disconnected");
return;
}
static unsigned waited;
int avail = wifi.available();
if (!avail) {
++waited;
return;
}
Serial.print("waited: ");
Serial.print(waited);
waited = 0;
Serial.print("\tto-read: ");
Serial.print(avail);
constexpr int max_read = 5120; // arbitrary chosen, takes effect sometimes
if (avail > max_read) {
avail = max_read;
Serial.print("\ttry-read: ");
Serial.print(avail);
}
uint8_t buf[avail + 1];
avail = wifi.read(buf, avail);
buf[avail] = 0;
Serial.print("\tdid-read: ");
Serial.println(avail);
Serial.println(reinterpret_cast<char *>(buf));
}
void loop() {
checkWiFiIncoming();
}
Skipping the actual content, the data available coming in on my R4 WiFi (don't have a GIGA) looks like this; starting with dozens of loop iterations waiting for a response:
waited: 73 to-read: 1412 did-read: 1023
waited: 0 to-read: 4625 did-read: 1023
waited: 0 to-read: 6546 try-read: 5120 did-read: 1023
waited: 0 to-read: 5523 try-read: 5120 did-read: 1023
waited: 0 to-read: 4500 did-read: 1023
waited: 0 to-read: 3477 did-read: 1023
waited: 0 to-read: 2454 did-read: 1023
waited: 0 to-read: 5763 try-read: 5120 did-read: 1023
waited: 0 to-read: 6176 try-read: 5120 did-read: 1023
waited: 0 to-read: 6589 try-read: 5120 did-read: 1023
waited: 0 to-read: 5566 try-read: 5120 did-read: 1023
waited: 0 to-read: 4543 did-read: 1023
waited: 0 to-read: 4956 did-read: 1023
waited: 0 to-read: 5369 try-read: 5120 did-read: 1023
waited: 0 to-read: 4346 did-read: 1023
waited: 0 to-read: 3323 did-read: 1023
waited: 0 to-read: 2300 did-read: 1023
waited: 0 to-read: 2414 did-read: 1023
waited: 0 to-read: 1391 did-read: 1023
waited: 0 to-read: 368 did-read: 368
It reads at most 1K at a time. Depending on how big a response you need to see all together at once in a single string, it may require some shuffling. Or maybe you can pass on the response in chunks to whatever needs it next. If it's really small, maybe you can get it in a single read.
Depending on which HTTPClient you're using, you might be able to use that to make the request, and then take over reading the response. Although requests are generally pretty simple. Responses, which might use Transfer-Encoding: chunked for example, can get complicated.
I have this function
int get_spotify_data()
{
WiFiClient client;
client.connect("XXXXX", 80);
client.println("GET /?operation=get HTTP/1.1");
client.println("Host: "XXXXX");
client.println("Connection: close");
client.println();
bool headers = true;
while (client.connected() && headers)
{
String line = client.readStringUntil('\n');
if (line == "\r")
{
headers = false;
}
}
String responseBody = "";
while (client.available())
{
responseBody += client.readString();
}
client.stop();
char* json = (char*)malloc(responseBody.length() + 1);
if (json) {
strcpy(json, responseBody.c_str());
}
if (json == "No current song" || json == "Error")
{
update_song_info("{'type': 'None','volume': 100,'time': 0,'duration': 0,'name': 'No current song','artist': 'No author'}");
}
else
{
update_song_info(json);
}
}
And these 3 second is from calling this function to " client.stop();"
I just tried your code and I don't have any difference
Before getting to the main issue; the following will not work:
for a few reasons
- If the API returns JSON, the word by itself,
Error, is not JSON. Compare these in your browser's JavaScript console
A string in JSON must have double quotes. So at a minimum, it would have to be> JSON.parse('Error').length SyntaxError: JSON Parse error: Unexpected identifier "Error" > JSON.parse('"Error"').length 5 > JSON.parse("'Error'").length SyntaxError: JSON Parse error: Single quotes (') are not allowed in JSON"\"Error\""and"\"No current song\"" - Most JSON APIs always return either an object in curly braces
{ }or an array in square brackets[ ], e.g.
not a bare string{ "error": "this is the error message" } - In C, comparing a
char *for a newly-allocated block of memory to a hard-coded string will always be not-equal. It's comparing two pointers, even if the text at those pointers is identical. Instead, you can compare aString:responseBody == "\"Error\"", which works as expected; and then to proceed further, make a copy to pass it on. (Be sure to properlyfreethat block when appropriate. Do you even need to make a copy?)
The main issue
I took most of your code, added some instrumentation, and applied it to what I had earlier. Your URL is redacted, so I switched the Google host to get a shorter payload (this may matter). All the changes are in setup
void setup() {
Serial.begin(115200);
WiFi.begin(SECRET_SSID, SECRET_PASS);
for (int i = 0; WiFi.status() != WL_CONNECTED; i++) {
Serial.print(i % 10 ? "." : "\n.");
delay(100);
}
Serial.println();
Serial.println(WiFi.localIP());
if (client.connect("google.com", 80)) {
hostConnected = true;
client.println("GET / HTTP/1.1");
client.println("Host: google.com");
client.println("Connection: close");
client.println();
}
auto start = millis();
bool headers = true;
while (client.connected() && headers) {
String line = client.readStringUntil('\n');
if (line == "\r") {
headers = false;
} else {
Serial.print(millis() - start);
Serial.print('\t');
line.trim();
Serial.println(line);
}
}
// while (hostConnected) {
// checkWiFiIncoming();
// }
// Serial.println(millis() - start);
String responseBody = "";
while (client.available()) {
int avail = client.available();
String part = client.readString();
Serial.print('\t');
Serial.print(part.length());
Serial.print("\t");
Serial.println(part);
Serial.print(millis() - start);
Serial.print('\t');
Serial.print(avail);
Serial.print('\t');
Serial.print(responseBody.concat(part) ? "added" : "didn't add");
Serial.print('\t');
Serial.println(responseBody.length());
}
client.stop();
}
This prints
122 HTTP/1.1 301 Moved Permanently
126 Location: http://www.google.com/
129 Content-Type: text/html; charset=UTF-8
137 Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-0jX11fd4RyoH1Xp6ua0X_Q' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
159 Date: Tue, 14 Jan 2025 03:20:17 GMT
163 Expires: Thu, 13 Feb 2025 03:20:17 GMT
167 Cache-Control: public, max-age=2592000
171 Server: gws
173 Content-Length: 219
175 X-XSS-Protection: 0
177 X-Frame-Options: SAMEORIGIN
181 Connection: close
219 <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
1209 219 added 219
host disconnected
It took 122ms to get the initial response line with the 301 status code. Another 60ms to get the rest of the headers, and then just over a second to read 219 bytes in the body in a single part. This is due to how readString works: unlike readStringUntil, the only way it can stop is due to a timeout with no-more-characters, which defaults to one second.
Now note the four commented-out lines in the middle. Uncomment them:
while (hostConnected) {
checkWiFiIncoming();
}
Serial.println(millis() - start);
to use the existing function in my sketch that does not use readString, so now a run looks like this
181 HTTP/1.1 301 Moved Permanently
185 Location: http://www.google.com/
189 Content-Type: text/html; charset=UTF-8
196 Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-he0o3-zgUrxPyVA1z77EnQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
218 Date: Tue, 14 Jan 2025 03:38:08 GMT
222 Expires: Thu, 13 Feb 2025 03:38:08 GMT
226 Cache-Control: public, max-age=2592000
230 Server: gws
232 Content-Length: 219
234 X-XSS-Protection: 0
237 X-Frame-Options: SAMEORIGIN
240 Connection: close
waited: 0 to-read: 219 did-read: 219
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
host disconnected
277
Just over a quarter-second total:
- The response starts 60ms later than before, which is expected to vary
- Still takes 60ms to read the headers, one at a time, and print them
- But now it only takes 37ms to read the remaining body, in one part/call
You're going to have to clarify. Using readString takes an extra second. By avoiding it, instead of 3 seconds, it could be two. But that could still be considered blocking for too long. Running the instrumented code, you can see where the delays are.
As I show, you can call out to a response-reader from the main loop function, so that whatever else you're doing can still run in the meantime.
- Waiting for the first byte back would take "no time"
- Each header line read separately, so you can act upon them, would take around 10ms each
- By limiting the amount that is read, each part of the body could be maybe 50ms or less.
What does a valid JSON response look like? How big is it?
1 Like