Chunked POST request

Using Arduino Programming Questions
#1

This is what my chunked POST request looks like. I am trying to send chunks of data from an SD card on the arduino to a server through a chunked POST request. Need chunked as it will allow me to send the data which will not all fit into the arduino's memory at once.

POST /upload HTTP/1.1
User-Agent: Arduino
Host: ooboontoo
Accept: /
Transfer-Encoding: chunked

25
this is the text, of this file, wooo!
1d
more test, of this file, fuck
0

However I am getting an error when the server tries parsing the request:

{"25\r\nthis is the text, of this file, wooo!\r\n"=>nil} Invalid request: Invalid HTTP format, parsing fails.
I tried reading the chunked requests documentation but it appears there are multiple ways of doing the same thing so I am a bit confused on what is the correct way.

Any advice would be appreciated thanks!

EDIT 2:

Here is the C code I wrote. I'm using the Arduino CC3000 client library for my connection to the server.

    String header = "POST /upload HTTP/1.1\r\n";
    header += "User-Agent: Arduino\r\n";
    header += "Accept \*/\*\r\n";
    header += "Transfer-Encoding: chunked\r\n\r\n";
    
    char hbuf[header.length() +1];
    header.toCharArray(hbuf, header.length() +1);
    
    String testfile = "this is the text, of this file, woo!\r\n";
    
    char testbuf[testfile.length() +1];
    testfile.toCharArray(testbuf, testfile.length() +1);
    
    String testagain = "more test, of this file, fuck\r\n";
    
    char againbuf[testagain.length() +1];
    testagain.toCharArray(againbuf, testagain.length() +1);
    
    String lenone = String(testfile.length(), HEX);
    String lentwo = String(testagain.length(), HEX);
    
    char buflenone[lenone.length() +1];
    lenone.toCharArray(buflenone, lenone.length() +1);
    char buflentwo[lentwo.length() +1];
    lentwo.toCharArray(buflentwo, lentwo.length() +1);
    
    Serial.println(buflenone);
    Serial.println(buflentwo);
    
    client.fastrprint(hbuf);
    client.fastrprintln(buflenone);
    client.fastrprint(testbuf);
    client.fastrprintln(buflentwo);
    client.fastrprint(againbuf);
    client.fastrprint("0\r\n");

EDIT 1:

here is the bash code I wrote to output the above request to telnet run with: ./testit.sh | telnet

#! /bin/bash
#testit.sh
#Arduino Telnet HTTP POST tests

header="POST /upload HTTP/1.1\n"
header+="User-Agent: Arduino\n"
header+="Host: localhost 3000\n"
header+="Accept: */*\n"
header+="Transfer-Encoding: chunked\n"

thisfile="this is the text, of this file"
thisfilelen=${#thisfile}
printf -v hexlen '%x' $thisfilelen
thisfile2="more test, of this file, fuck"
thisfilelen2=${#thisfile2}
printf -v hexlen2 '%x' $thisfilelen2
end="0"


echo "open localhost 3000"
sleep 2
echo -e $header
echo -e $hexlen
echo -e $thisfile
echo -e $hexlen2
echo -e $thisfile2
echo -e $end
sleep 2

Here is where I asked the question on stack overflow:

I've spent hours and hours and hours trying to figure out the proper way to send a chunked post request but no resource seems to be useful(and consistent).

#2

Also as a note:

the code is in bash for debugging purposes (as running and debugging the code on the Arduino was taking forever). I will have no issues in transferring over the code to C, right now I am just very confused on how to do a chunked HTTP post request properly

#3

bumping for inactivity

#4

grantnicholas:
bumping for inactivity

Perhaps you should ask your question on a "the code is in bash" forum. Doesn't sound like an arduino issue.

#5

I was trying to isolate the problem by specifically asking for help on the HTTP POST request portion of the problem, instead of possibly introducing small C errors with newlines etc, but I see where you are coming from so I added the C code I wrote as well.

I am still woefully stuck so any help would be greatly appreciated.

#6

I’m not understanding why you have to “chunk” the code. It doesn’t have to all fit into RAM at one time. Just write it out to the server directly from wherever it is stored. Moving it first to RAM is a waste of time. Worst case this might require making a simple modification to whatever the “client” class is, but you’d end up with a much more efficient system.

Regards,
Ray L.

#7

RayLivingston:
I'm not understanding why you have to "chunk" the code. It doesn't have to all fit into RAM at one time. Just write it out to the server directly from wherever it is stored. Moving it first to RAM is a waste of time. Worst case this might require making a simple modification to whatever the "client" class is, but you'd end up with a much more efficient system.

Regards,
Ray L.

What does "just write it out to the server directly from wherever it is stored" mean. The data is from a datalogger that will be storing CSV data into a (very large) file and then periodically POSTing that data to a server. How do I send all the data in a POST request without putting it into main memory, because as far as I know the only way to do that would be opening the file and then writing the file contents in the POST request (specifying a content length of the size of the POSTed data).

And to specify this content length we would need to have all the content in main memory. IE) using a chunked request allows us to send the data in small chunks which we can keep in main memory and thus we can specify the content lengths of the chunks instead of the whole block of data

#8

grantnicholas:
What does “just write it out to the server directly from wherever it is stored” mean. The data is from a datalogger that will be storing CSV data into a (very large) file and then periodically POSTing that data to a server. How do I send all the data in a POST request without putting it into main memory, because as far as I know the only way to do that would be opening the file and then writing the file contents in the POST request (specifying a content length of the size of the POSTed data).

And to specify this content length we would need to have all the content in main memory. IE) using a chunked request allows us to send the data in small chunks which we can keep in main memory and thus we can specify the content lengths of the chunks instead of the whole block of data

If the data is stored on a disk, what I would do, is modify the client code so I could simply pass it a file handle, and it would send the data from the file, by reading a line/block/whatever at a time, sending it, then repeating until the entire file was sent. That’s basically the approach I used in the WebServer I wrote for the Due.

If the file is “very large”, there is an argument for breaking it up for the sake of error detection/recovery, but I don’t know if that’s necessary in your case. Why not send the data more often?

Regards,
Ray L.

#9

I would get the SD file size and use that as the Content-Length in the header, then send it 64 bytes at a time until it is all sent.

I haven't tried it yet, but it sounds like it would work if the file size returned is correct.

#10

When you say send the data 64 bytes at a time, do you mean making a new post request for each 64 bytes of data being sent?

#11

No. Send it as one POST request. I use 64 bytes for two reasons.

  1. It is small enough that it doesn't eat up too much SRAM on an Uno,
    and
  2. It is smaller than the Wifi shield can send in one packet (90 bytes).

The file size is accurate. I just checked it with a couple of photos I download with my web server. It sends the pics 64 bytes at a time. Like this:

int clientCount = 0;
char tBuf[65];

File myFile = SD.open(fileName);

if(!myFile) {
  Serial.println(F("open error"));
}
else {
  // send the first part of your request header here
  // get your "Content-Length" here
  unsigned long fileSize = myFile.size();
  // send the Content-Length and the rest of your request header here

  Serial.print(F("size = "));
  Serial.print(fileSize);
  Serial.println(F(" bytes"));

  // send the POST data 64 bytes at a time here
  while(myFile.available()) {
    tBuf[clientCount] = myFile.read();
    clientCount++;
    tBuf[clientCount] = 0;

    if(clientCount > 63) {
      client.write((byte*)tBuf,64);
      clientCount = 0;
    }

  }

  // send any remaining bytes
  if(clientCount > 0) {
    client.write((byte*)tBuf,clientCount);
  }

  // close the file
  myFile.close();
}
#12

Gotcha! Awesome I'll try to get that to work. It is really frustrating me however that the chunked http request is not working because that is exactly what the chunked transfer encoding is designed for.

#13

You can send it as chunked (edit: using clientCount in the example above). Must be HTTP/1.1 and follow a strict protocol.

You know the length of the file tho.

#14

Thanks SurferTim your method of reading the byte size from the file worked well!

For future people looking for help, I’ll attach the entire sendData function I made [which uses the adafruit CC3000 client library and the SD card library to send data stored in a file to a webserver using an HTTP POST request.

bool sendData() {

  Serial.println("connecting...");
  
  uint32_t ip = 0;

  while (ip == 0){
    if (! cc3000.getHostByName("www.yourserver.com", &ip)){
     Serial.println(F("Couldn't resolve!"));
    }
    delay(500);
  }
  
  Serial.println(ip);
  delay(1000);
  Adafruit_CC3000_Client client = cc3000.connectTCP(ip, 80); //3000
  delay(1000);

  if(client.connected()){
      Serial.println("connected");

      String fileName = "fileonsd.txt";
      char bufff[fileName.length() +1];
      fileName.toCharArray(bufff, fileName.length() +1);
      File myFile = SD.open(bufff);
      
      if(!myFile) {
        Serial.println(F("open error"));
      }
      else {
        
        int fileSize = myFile.size();
        
        Serial.print(F("size = "));
        Serial.print(fileSize);
        Serial.println(F(" bytes"));
        
        // send the Content-Length and the rest of your request header here
        client.fastrprintln(F("POST /upload HTTP/1.1"));
        client.fastrprintln(F("Content-Type: text/plain"));
        client.fastrprintln(F("Host: yourhosthere"));
        client.fastrprintln(F("Cache-Control: no-cache"));
        client.fastrprint(F("Content-Length: ")); 
        client.println(fileSize);
        client.println("");
        
        
        //use a 1 byte "buffer" instead of an array --> makes code simpler
        //user SurferTim's method if performance is an issue as I think sending more client.writes may
        //be an issue with lots of data

        byte b;
        while(myFile.available()){
          b = myFile.read();
          client.write(b);
          Serial.write(b);
        }
        client.fastrprintln("");

        //capture the response from the server; should be 200 if everything went well
        char c = 0;
        int q = 0;
        while ( q < 50 || client.available() > 0 ) {
                c = client.read();
                q++;
                Serial.print( c );
        }

       
        // close the file
        myFile.close();
      }
      
      
    
    return true;
  } 

  else {
    // you didn't get a connection to the server:
    Serial.println("connection failed");
    return false;
  }
  
  
}
#15

Hi grantnicholas,

I know this is a very old thread but it seems to be the only one covering "using POST to send a file from SD card to a server".

I trying to use your example from your last post but I keep receiving a 400 bad request.

Did that happen to you too while working on this?

Cheers

#16