WiFiNINA POST body max size

Official Hardware Nano Family Nano RP2040 Connect
1

I'm trying to make an HTTP POST request to a server with a large binary payload (100kb). I cannot make this work because the client fails to write more than ~6000 bytes.
Here are the experiments I did:

#define NUM_SAMPLES 32000

int16_t samples[NUM_SAMPLES];
WiFiClient client;

// ...other parts omitted...

// no body, goes ok
client.println("POST /api HTTP/1.1");
client.println("Host: salernosimone.com");
client.println("Connection: close");
client.println("Content-Type: application/octect-stream");
client.println();

// short body, goes ok
client.println("POST /api HTTP/1.1");
client.println("Host: salernosimone.com");
client.println("Connection: close");
client.println("Content-Type: application/octect-stream");
client.println("Content-Length: 4096");
client.println();
client.write((uint8_t*) samples, 4096);

// large body, single write, not received by the server
client.println("POST /api HTTP/1.1");
client.println("Host: salernosimone.com");
client.println("Connection: close");
client.println("Content-Type: application/octect-stream");
client.println("Content-Length: 8192");
client.println();
client.write((uint8_t*) samples, 8192);

// large body, chunked write, still not received by the server
client.println("POST /api HTTP/1.1");
client.println("Host: salernosimone.com");
client.println("Connection: close");
client.println("Content-Type: application/octect-stream");
client.println("Content-Length: 8192");
client.println();
client.write((uint8_t*) samples, 4096);
client.write((uint8_t*) samples, 4096); // this actually writes < 2000 bytes

Anyone knows what's going on? How can I make the request succeed?

2

but does it send it all at the end?

3

No. If the content length is < 6000, the request goes fine and I get a response from the server. Otherwise, the request does not complete.

I guess it is because I set Content-Length=100k, but only send ~6k bytes, so the NGINX server discards the request as malformed (or something in the like).

4

why is there uploading * 2?

5

I made an error in the first code. ContentLength == length(samples). Since samples is made of uint16_t values, each value is made up of 2 bytes. I'm looping i from 0 to number of samples, thus the data I have to upload is 2 * the current chunk size.

That said, you're not suggesting any improvement so far.

6

what I say is that the library is OK. you have some error in the sketch. and I feel those length counters don't match

7

I edited the first message with the experiments I did. The counters are not the problem.

8

write returns size_t, so you can check how much was actually written. That's the basic contract for these low-level read and write functions: "try this many; how many got through?" You have to do the math accordingly. For example, WiFiNINA doesn't support writing a Stream, but here is how ESP32 does it

size_t WiFiClient::write(Stream &stream)
{
    uint8_t * buf = (uint8_t *)malloc(1360);
    if(!buf){
        return 0;
    }
    size_t toRead = 0, toWrite = 0, written = 0;
    size_t available = stream.available();
    while(available){
        toRead = (available > 1360)?1360:available;
        toWrite = stream.readBytes(buf, toRead);
        written += write(buf, toWrite);
        available = stream.available();
    }
    free(buf);
    return written;
}

That loop works without a delay between each block, but you might need one.

If the buffer is around 6000, then that makes sense. And you're setting Content-Length, so the server will wait until it gets that much. If you're short, that will timeout.

9

You restated the problem, which I already know. But you did not suggested a solution.
How do I empty the internal buffer, if that's the problem?
Even considering the actually written bytes (e.g. 2000 in the second call), if I write a third time it returns 0.

10

I suggested you use a loop. The example I pasted is actually a little optimistic, but the idea is the same

  • you know exactly how many bytes you want to send
  • a uint8_t pointer starts at the beginning of the samples, obviously
  • try to write the whole thing; or more accurately, whatever is left
  • however many bytes got written, you can advance the pointer that much
    • if zero bytes are written, that's not a special case. The pointer does not advance, and you try again
  • I also suggested you might use a (small) delay after each write. It may be either
    • unnecessary: a dedicated chip or extra core/task empties the buffer "over the wire"
    • more efficient: instead of repeatedly pushing zero or a few bytes, you can yield while the background work is happening
    • required: you have to yield to the background work
    • insufficient: some extra step is missing

I don't have an RP2040, but given the product description it is more likely one of the first two, not the last two.

11

A mixture of chunkSize and delay did the job. Depending on how much data you want to send, you have to configure those 2 values.

E.g.

8k body => chunkSize = 2048 + delay 10ms works fine
70k body => same config doesn't work
70k body => chunkSize 2048 + delay 50ms works fine (most of the times)

For future reference, here's the code I used to send 130k of data.

const uint16_t chunkSize = 2048;
uint8_t *body = (uint8_t*) samples;
size_t remaining = count * 2;
size_t offset = 0;
size_t timeout = millis() + 10000;

while (millis() < timeout && remaining > 0) {
  const uint16_t writing = min(chunkSize, remaining);
  const uint16_t written = client.write(body + offset, writing);
  Serial.print("Written ");
  Serial.print(written);
  Serial.print(" out of ");
  Serial.print(writing);
  Serial.println(" bytes");
  delay(50);

  remaining -= written;
  offset += written;

  if (written == 0) {
    Serial.println("Error");
    break;
  }
}