HTTP GET returns only partial page (WiFiClientSecure / ESP32)

Hello,

What I am trying to achieve
I want to use an ESP32 and a MAX7219 display to show various data that I pull from the internet. I have succesfully downloaded a json file. Some data is not available as json, so I want te be able to get information from any website. I have an issues with a particular website when using WiFiClientSecure on an ESP32.

My problem
The website returns only the beginning of the HTML source code. Viewing the source code from my browser (Firefox) or downloading the page with WGET does work and gets me the information that I need.

The code

    client.print("GET /foo/bar.aspx HTTP/1.1\r\nHost: www.somesite.com\r\n\r\n");
    while (client.connected()) {
      String line = client.readStringUntil('\n');
      if (line == "\r") {
        Serial.println("headers received");
        break;
      }
    }
    // if there are incoming bytes available
    // from the server, read them and print them:
    while (client.available()) {
      char c = client.read();
      Serial.write(c);
    }
    client.stop();

Results
This gets me the beginning of the source (about 140 lines). The last thing I get is the part of a very long string. (Is this a clue?)

<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUK.......

I am currently researching HTTP headers, but this is taking more time than I would like.

Help is appreciated.

I now think I might need recursive downloading. Is this possible with HTTP GET?

I could always set up a Raspberry that handles the WGET for the ESP32. That will work, but is kinda silly. :confused: