GET requests - stray stuff in answers

Projects Networking, Protocols, and Devices
1

I am working on a system that collects information about Dynamic Electricity Prices from a website Entsoe
The answer has info for 2 days, one price per quarter of an hour. That's 192 lines and with a lot of headers and stuff, all in HTML, it results in a String of about 16K.
When I put the identical GET request in the search line of FireFox or Safari I get a nice HTML answer.
However when I do this request in code on a Nano ESP32, I get almost the same, but spread across the answer, some pesky inserts are added. They all correspond to the HEX number format, although meaningless and very difficult to filter them out with regexp.

I asked the people of Entsoe, they are not aware of anything that inserts HEX numbers or whatever in the requests and certainly not with differences depending on what system does the request.
Here some examples:
76<price.amount>94.97</price.amount>15d015<price.amount>107.56</price.amoun2c0dt>ff847<price.amount>18.13</price.amount>

These things appear anywhere in the HTML text.
With an identical request, seconds apart, the inserts are different and on different places.
May be browsers like FireFox get them as well but manage to filter the out.
I would be grateful if I could get a hint on how to solve this.
I manage to send these GET requests with a Nano ESP32 but not with a Uno R4 WiFi but that's another story.

2

My HTML book is put away in the garage so I can't look. What does your HTML documentation say is the possible answer?

3

Can you share the GET request that you are making ?

4

URL encoding would usually precede the encoded hex value of the byte with a % character. Is this being saved to a file in LittleFS or being printed directly to the serial monitor?

Could you share the code Also would be interested to see the request.

5

That's likely Transfer-Encoding: chunked, which you can see in the response headers.

A proper HTTP client will handle that for you. But it's not too difficult to do it manually.

It is supported in HTTP 1.1, but not in 1.0. So if you change the request to end with HTTP/1.0 -- those bytes actually mention something -- that might "fix" it. Or the server can refuse.

6

Here is the request. I let out my personnel token. I guess that is the way to do it.

"GET http://web-api.tp.entsoe.eu/API?documentType=A44&in_Domain=10YBE----------2&out_Domain=10YBE----------2&offset=0&contract_MarketAgreement.type=A01&securityToken=M Y T O K EN &periodStart=202607020000&periodEnd=202607032359 HTTP/1.1"

7

I'm not 100% sure, especially since I don't know the service's API calls (and we can't test it without a token). But in the meantime, please always include your instructions, code, and response text in the "CODE" tag so we can read the text "as is" and not potentially "interpreted" by the forum.

But besides the GET format, to complete this topic, what else do you send in the request? Are there any headers? Can you post everything you send to the server in its entirety (in a "CODE" tag, removing your token), from the connection to its response?

In all my years, I've never seen server responses with extraneous alphanumeric (apparently hexadecimal) strings, and I don't think any GET, even a more or less correct one, could do that. So, since requests made with Firefox return the expected response, I suspect something is wrong with the code (which we can't see), perhaps in the handling of response strings, or in some unsuitable or obsolete library.
Without further information, I don't think I can offer any helpful advice.

8

Was thinking along those lines as well. The spurious apparent HEX is just being injected into the text even in the middle of a word.

9 

```cpp

#define myTitle "WiFiClientSecure_NanoEsp32"
#include <WiFiClientSecure.h>
#include <Arduino.h>
const char* ssid = "Proximus-Home-2508_EXT";
const char* password = "wpzb9prjx2bu2";
int keyIndex = 0;
const char* server = "web-api.tp.entsoe.eu";  // Server URL
String EnstoeAnswer = "";
//
WiFiClientSecure client;


void setup() {
  Serial.begin(115200);
  delay(1000);
  Serial.println(myTitle);
  Serial.println(__FILE__);
  Serial.println(ssid);
  Serial.println(password);

  Serial.print("Attempting to connect to SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  // attempt to connect to Wifi network:
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    // wait 1 second for re-trying
    delay(1000);
  }

  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.println("\nStarting connection to server...");
  client.setInsecure();  //skip verification
  if (!client.connect(server, 443))
    Serial.println("Connection failed!");
  else {
    Serial.println("Connected to server!");
    // Make a HTTP request:
    client.println("GET http://web-api.tp.entsoe.eu/API?documentType=A44&in_Domain=10YBE----------2&out_Domain=10YBE----------2&offset=0&contract_MarketAgreement.type=A01&securityToken=c67dd299-4461-4106-ad5b-836c30efc603&periodStart=202606170000&periodEnd=202606172359 HTTP/1.1");
    client.println("Host: web-api.tp.entsoe.eu");
    client.println("Connection: close");
    client.println();

    while (client.connected()) {
      while (client.available()) {
        unsigned long characterCount = 0;
        char c = client.read();
        characterCount++;

        EnstoeAnswer += c;
        characterCount++;
      }
    }
    //With this one returns, bewlines, spaces... are filtered out, saving memory.  This way, the first
    // chunk token, the HRX number.
    // if (c != '\n' && c != '\r' && c != 0x20 && c != 0x22 && characterCount >370){}
    //EnstoeAnswer = client.readString(); //both statements yield same resul
  }


  Serial.println(EnstoeAnswer);
  Serial.print("Entsoe Answer length = ");
  Serial.print(EnstoeAnswer.length());
}
void loop() {
  // do nothing
}
Thanks everybody for your replies.

I have never been diving deep into the HTTP stuff, so I was not aware of that chunk thing.

By receiving this long answer (±30k) I filtered out the returns, newlines, spaces etc...to save some memory. This way I. was not aware of the appearance of the chunk labels.

How would we proceed to re-concatenate the answer? By counting the characters as indicated by the first label, then expecting the next label and so on...?

It was written somewhere that a decent HTTP client can handle this.  The code herewith is derived from the example WiFiClientSecure.  At fist sight nothing is provided to de-chink incoming HTTP files.

Or if anybody could provide a link to something that explains / solves this little problem, I would be grateful.

Herewith the code with my keys inside. It works on a Nano ESP32.  I need to manage to make it work on a Uno R$ WiFi because the libraries that are called are not the same.

Anyway thanks. I learned that I still have to learn a lot.

I hope the sketch is well attached. If not drop me a line.
10

I tested with curl and it worked: the server delivered the goods without Content-encoding: chunked.

11

Using the code from post #9, on a Wemos D1 R32 clone, I can duplicate your problem with the extraneous characters throughout the code. Changing to HTTP/1.0 appears to eliminate the problem. Might be something in the ESP32 that is causing the data to be sent chunked.

Might be a good idea to reserve the space for the String, since it is so large, although that seems to have no effect on the output.

12

Well, this is certainly eager to debug when you post your auth token; now that you've done that on a public forum, you should invalidate it as soon as possible, and change it -- but maybe after you're done needing help.

The bigger issue is that R4 WiFi only has 32KB of RAM. An empty sketch uses 20% of that. Not enough left for the 29KB payload, which is XML btw, not HTML. That's evident in the response headers

Content-Type: text/xml
Transfer-Encoding: chunked

Not only is XML more verbose, it's also pretty-printed with spaces (tabs would be fewer bytes)

              <Point>
                <position>95</position>
                  <price.amount>138.24</price.amount>
              </Point>
              <Point>
                <position>96</position>
                  <price.amount>133.78</price.amount>
              </Point>
          </Period>
      </TimeSeries>

In contrast, getting the data as JSON -- which is not pretty-printed -- is a lot more compact. Same fragment:

"94":[138.24],"95":[133.78]}}

The size goes from 29KB down to 4KB. To do that, you need an Accept request header

    client.println("GET /API?lots=of&query=params HTTP/1.0");
    client.println("Host: web-api.tp.entsoe.eu");
    client.println("Accept: application/json");
    client.println();

Using HTTP/1.0 also means you don't need Connection: close -- connections are not reused with 1.0. And the GET request line does not contain the scheme://host:port part; it starts with the slash that starts the URL-path. DNS has already gotten you to the right IP address in the client.connect(server) call; and the Host header, optional in HTTP 1.0 (required in 1.1), will direct you to the correct server instance when an IP address hosts multiple servers.

4KB is small enough that reading a single giant string might work. There might even be enough memory to use a library like ArduinoJson. But without the chunks and the XML tags, it might be easy enough to read the stream directly "byte by byte" with a state machine.

13

Thanks for the advice kenb4.
I will continue with json.
The server responds all the time with HTTP/1.1 ... regardless if the GET request is terminated with 1.1 or 1.0.
May be the messages will arrive not chunked because they are a lot shorter.
The Entsoe key is changed already.

14

The server responds with the "best" version it supports, in case the client wants to upgrade. It omits the 1.1 features like chunked when responding to a 1.0 request. (That's easy enough with HTTP v1.x -- more complicated with HTTP v2 in the mix.)

Note that one advantage of chunked is that you can clearly tell when the response is done: it sends a final zero-sized chunk (with nothing after/in it). It's appropriate when the web server is passing along some data without knowing the size in advance, like the API call in this case.

When the server is sending something known, like an image or CSS file, it normally includes a Content-Length response header. The advantage in that case is that exact size can be allocated in advance, which is definitely relevant when operating in a constrained environment like an Arduino.

When the client says "I only want HTTP/1.0", in this case the server does neither chunked nor Length. It just sends the bytes as it gets them and closes the connection when done. At least with content types like JSON and XML, you can tell whether you got the entire document. For most JSON, did you get the final } that matches the { at the start?

15

Just a small addition to the HTTP version topic (I've never dealt with "blocky" responses, so I can't help more than "kenb4"):

The sketch is ok, but you should move your description (from "Thanks everybody.." to the end) outside the "code" block (I mean after the final "triple quote", because your comments are formatted together like the code and are difficult to be read.