https.getString(); returns empty string

Projects Networking, Protocols, and Devices
1

I've googled this and lots of people seem to have problems with this, but nowhere have I found a solution. I have a NodeMcu V3 ESP8266. I'm trying to read a page, which is generated by a PHP script. The page looks fine in a browser. It outputs something like this:

Das klinget so herrlich 1: 0, 93, 80,
2: -116, 0, 0,
3: 240, 91, 80,
4: 480, 74, 80,
5: 480, 78, 80,
6: 480, 81, 80,
7: 480, 90, 80,
8: 960, 98, 80,
9: 1440, 102, 80,
10: 1920, 93, 80,
11: 2160, 90, 80,
12: 2400, 81, 80,
13: 2400, 85, 80,

Plain text. Sometimes https.getString(); returns the right content, but mostly it returns nothing. Typical output in my Serial windows is:

[HTTPS] begin...
[HTTPS] GET...
[HTTPS] GET... code: 200

Wait 10s before next round...

After the "code: 200" it should show the data listings, before the "Wait 10s...". I've tested several code examples, which should work with https. This is a part of my code:

    HTTPClient https;

    Serial.print("[HTTPS] begin...\n");
    if (https.begin(*client, jigsaw_host, jigsaw_port)) {  // HTTPS

      Serial.print("[HTTPS] GET...\n");
      // start connection and send HTTP header
      int httpCode = https.GET();

      // httpCode will be negative on error
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          String payload = https.getString();
          Serial.println(payload);
        }
      } else {
        Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
      }

      https.end();
    } else {
      Serial.printf("[HTTPS] Unable to connect %d\n");
    }
  }

It hardly ever reaches the "failed, error" part, which means the GET() seems to work. Only the getString(); fails. I'm willing to test a completely different approach, if I only knew what. The getString() reads the whole page in one run, right? Has the failure anything to do with the size of the data chunk? Is there a way to read a web page line by line?

2

try this:

instead of

        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          String payload = https.getString();
          Serial.println(payload);
        }

do

        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
           const String& payload = http.getString(); // get a reference instead of duplicating
          Serial.println(payload);
        }
1 Like
3

That helped a little. But it seems the ESP8266 still can't take the whole payload. This is my simple php script:

<?php

    echo "<html>\n";
    echo "<head>\n";
    echo "</head>\n";
    echo "<body>\n";
    
    echo "<h1>Hi</h1>\n";
    $j = 1;
    for ($i = 1; $i <= 500; $i++)
    {
        for ($k = 1; $k <= 10; $k++)
        {
            echo $j . "  ";
            $j++;
        }
        echo "<br/>\n";
    }
    
    echo "</body>\n";
    echo "</html>\n";
    
?>

So it should print 500 rows, 10 numbers per row, up to 5000. It stops at 2646:

...
2621  2622  2623  2624  2625  2626  2627  2628  2629  2630  <br/>
2631  2632  2633  2634  2635  2636  2637  2638  2639  2640  <br/>
2641  2642  2643  2644  2645  2646  
Wait 10s before next round...
[HTTPS] begin...
[HTTPS] GET...
...

Is there really no way to read the stuff in smaller chunks?

4

Really if the answer can be very large your server should use Chunked transfer encoding - Wikipedia

5

Seems 20 kB is what my ESP8266 can handle at its most. Even then it might fail.

6

Do you really need all that data to be sent ?

7

Well, one download is one song, containing some 2000 note events. One note event contains a 32 bit time stamp, an 8 bit note number and 8 bit velocity. As a binary file, it would be some 12 kB. As a text file with csv, it's up to 30 kB.

8

As it does not work you probably need to plan for a Chunked response from your PHP script and handle the various packets on the arduino side

9

Yes, I'm working on it. But I don't know how to read the separate chunks. The library doesn't seem to have anything about chunks. I mean, I need to be able to read say 5 kB, then write it through serial to another board (which has all the memory and all the pins for the actual music production, but lacks wifi), then read the next 5 kB chunk and so on.

And actually I don't know either, how I do the server side stuff. I imagine I count the bytes I want to output, I collect a 5 kB chunk, then I write the chunk size in hex, cr lf, then I write the chunk itself

10

may be explore techniques used by ArduinoJSON

see How to use ArduinoJson with HTTPClient? | ArduinoJson 6

1 Like
11

I'm about to solve the problem, I think.
I got confused by the example code I used as a template. It does this:

std::unique_ptr<BearSSL::WiFiClientSecure>client(new BearSSL::WiFiClientSecure);

...where client is created. And later:

    HTTPClient https;
...
    if (https.begin(*client, "https://johanhalmen.link/rohrenspiel/get_next.php")) {  // HTTPS
...
        const String& payload = https.getString();

I kind of thought I only could access the data through the https object. And it only read the whole 30 kB chunk, no matter what I tried. Then I scratched the https.getString(); thing and used this instead:

          String a;
          do
          {
            a = client->readStringUntil('\n');
            Serial.println(a);
          }
          while (a.indexOf("0x7ff") == -1 && a.indexOf("&eof&") == -1);

So right now it reads one line of text and immediately outputs it. Every file, no matter if it is 5 kB or 35 kB, gets dumped as text to my serial monitor, line by line. Each file ends with a line containing 0x7fff. If no file is available, the PHP script outputs &eof&.
But now I get an output like this:

[HTTPS] GET...
[HTTPS] GET... code: 200
13b1

Mo Li Hua
0, 75, 64,
-60, 0, 0,
0, 91, 64,
120, 94, 64,
240, 82, 64,
240, 91, 64,
360, 89, 64,
480, 87, 64,
600, 89, 64,
720, 82, 64,
...

It is almost ok. The [HTTPS] lines I can get rid of. But the following hex number 13b1 and the following empty line is the chunked thing working. I don't need it, but I guess I can't turn it off. I have to read the hex value 0x13b1 and start counting 0x13b1 bytes starting with "Mo Li Hua", to recognise the end of this chunk and the beginning of next chunk.
I wrote the PHP script that outputs the file. But the underlying server system is the one adding the chunked data.
So right now I read one line from the PHP generated text file dump and output the line to Serial. For my project, I need to replace the Serial with another serial connection to another board (STM32U575-Nucleo), but that shouldn't be a problem. Since the chunked data is there, cutting automatically the output into some 5000 byte chunks, which my ESP8266 probably can handle, I might change the line-by-line dump to chunk-by-chunk dump.

12

Yes I think PHP will do that for you when the buffer is full

If I remember correctly The protocol on Unix sends on a separate line

\nSize_in_Hex\n

Or

\r\nSize_in_Hex\r\n

So in the binary flow the 0x10 or 0x13 you see are the carriage return or line feed bytes.

13

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.