Parse HTTP POST data from EthernetClient

Hi.

I'm looking for some help parsing an HTTP POST request from an EthernetClient.
I have an Ethernet2 shield connected to a Mega2560 and am using the Ethernet2 and TextFinder libraries.

A browser submits an HTML page containing an HTML form to my ethernet shield.
The form contains a single file upload element:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8">
        <title></title>
    </head>
    <body>
		<form action="http://192.168.100.150/" enctype="multipart/form-data" method="post">
			<input type="file" name="iniFileName" id="iniFileId">
			<input type="submit" value="Submit">
		</form>
    </body>
</html>

The ethernet shield receives the submitted data and i want to extract just the binary (uploaded file) data.
The HTTP request looks like this:

/ HTTP/1.1
Host: 192.168.100.150
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Type: multipart/form-data; boundary=---------------------------163482754226683
Content-Length: 378

-----------------------------163482754226683
Content-Disposition: form-data; name="iniFileName"; filename="network.ini"
Content-Type: application/octet-stream

my config file

[network]
dns = 192.168.100.1
gateway = 192.168.100.1
ip = 192.168.100.150
mac = 90-A2-DA-10-71-9F
port = 80
subnet = 255.255.255.0

-----------------------------163482754226683--

The uploaded file contents are:

my config file

[network]
dns = 192.168.100.1
gateway = 192.168.100.1
ip = 192.168.100.150
mac = 90-A2-DA-10-71-9F
port = 80
subnet = 255.255.255.0

It's an ini file that starts with a comment and ends with a trailing new line.
I want to extract the binary data which is my uploaded file and save it to an SD card in the ethernet shield.
I have a class which has a method and in this method i'm currently trying this:

boolean StorageManager::writeFile(EthernetClient ethernetClient){
	boolean success=false;

	TextFinder textFinder(ethernetClient);

	//	using 96 as max number of chars in the boundary value
	char boundaryValue[96];
	int charsRead=textFinder.getString("boundary=", "\r\n", boundaryValue, 96);
	if(charsRead>0){
		boundaryValue[charsRead]='\0';
		strcat(boundaryValue, "--");
		Serial.print("boundary: ");
		Serial.println(boundaryValue);

		//	max filename length is 8.3 filename + null terminator = 13 chars
		char filenameValue[13];
		charsRead=textFinder.getString("filename=\"", "\"", filenameValue, 12);
		if(charsRead>0){
			filenameValue[charsRead]='\0';

			Serial.print("filename: ");
			Serial.println(filenameValue);

			if(textFinder.find("octet-stream\r\n\r\n")){
				File file=SD.open(filenameValue, O_WRITE | O_CREAT | O_TRUNC);
				if(file){
					while(ethernetClient.available()){
						file.print((char) ethernetClient.read());
					}
					file.close();
					success=true;
				}
			}
		}
	}

	return success;
}

The code currently saves this data to my SD card:

lan2fsk config file

[network]
dns = 192.168.100.1
gateway = 192.168.100.1
ip = 192.168.100.150
mac = 90-A2-DA-10-71-9F
port = 80
subnet = 255.255.255.0

-----------------------------7280166718996--

My question is how to remove that trailing boundary value "-----------------------------7280166718996--"?
I could modify my while loop:

while(ethernetClient.available()){
	char c=(char) ethernetClient.read();
	if(some_condition){
		//	what condition can i test for to see if the final line "-----------------------------7280166718996--" has been reached?
	} else {
		file.print(c);
	}
}

I could create a char array buffer and read each line of binary data into the buffer then compare the buffer to the boundary value - this would tell me whether or not i want to write the buffer to the file.
And this is what i was about to start coding.

BUT...
This method will not only save text based ini files, it will also be used to upload and save binary hex files - i want to be able to upload my compiled sketch (after an update) to the shield and am then hoping to (remotely) flash the updated sketch from the SD card.
My sketch is currently around 30KBs in size when compiled.
Imagine a text file with a single line of 30,000 characters.
I wouldn't be able to read that single line into a buffer and compare it to my boundary value.

So the solution i was about to start coding would work when i upload a simple text ini file.
But it wouldn't be able to handle a large binary file.

Does that all make sense?

How can i get all data in the EthernetClient Stream up to but not including the last line - the line that contains my boundary value?

Thanks a lot for any help.

A browser submits an HTML page containing an HTML form to my ethernet shield.

No, it doesn't. The form contains an action field, triggered by any submit feature on the form. The browser uses the data that it collects and makes a GET or POST request to your server (if your server is defined in the action field).

	char boundaryValue[96];
	int charsRead=textFinder.getString("boundary=", "\r\n", boundaryValue, 96);
	if(charsRead>0){
		boundaryValue[charsRead]='\0';
		strcat(boundaryValue, "--");

You have an array that can hold 96 characters. You tell the getString() method that it can put up to 96 characters in the array. And, then you add three more. Do you regularly buy shoes three sizes too small, too?

How can i get all data in the EthernetClient Stream up to but not including the last line - the line that contains my boundary value?

If you quit using crutches (like the TextFinder class), you'll find that you can read a reasonable amount of data, like say 80 characters/bytes. If you haven't found the start of the data of interest (that is, if you haven't seen the boundary start marker), pitch the data in the bit bucket, and read some more.

If you have seen the boundary start marker, save the data to the SD card.

Then, the only challenge will be the boundary marker text spanning two records.

But, that is easily dealt with by reading and saving until the carriage return or line feed appears. The saved data either starts with Content-length, or it doesn't. If it does, determine the value (the length), and then read and store that number of characters. The boundary end marker doesn't really contribute much.

Thanks for the comments - you got me thinking along the right lines and i came up with this:

#define HTTP_BOUNDARY_MAX_LENGTH 70
#define WRITE_BUFFER_SIZE 128

boolean StorageManager::writeFile(EthernetClient& ethernetClient) {
	// TODO make parsing compatible with linux line endings
	// (currently only windows line endings are supported)
	boolean fileWritten = false;
	if (beginSucceeded) { // beginSucceeded is set elsewhere to true if SD.begin() returned true
		char boundaryKey[] = "boundary=";
		if (ethernetClient.find(boundaryKey)) {
			char tmpString[HTTP_BOUNDARY_MAX_LENGTH + 1]; // add one for null
			int readBytesCount = ethernetClient.readBytesUntil('\r', tmpString, sizeof(tmpString));
			tmpString[readBytesCount] = '\0';
			char boundaryValue[HTTP_BOUNDARY_MAX_LENGTH + 3] = "--"; // add three for null and --
			strcat(boundaryValue, tmpString);
			Serial.print("boundary: ");
			Serial.println(boundaryValue);

			char filenameKey[] = "filename=\"";
			if (ethernetClient.find(filenameKey)) {
				char filenameValue[13];
				readBytesCount = ethernetClient.readBytesUntil('"', filenameValue, 13 - 1);
				filenameValue[readBytesCount] = '\0';
				Serial.print("filename: ");
				Serial.println(filenameValue);
				char octetStreamKey[] = "octet-stream\r\n\r\n";
				if (ethernetClient.find(octetStreamKey)) {
					File file = SD.open(filenameValue, O_WRITE | O_CREAT | O_TRUNC);
					if (file) {
						char writeBuffer[WRITE_BUFFER_SIZE];
						while (ethernetClient.available()) {
							readBytesCount = ethernetClient.readBytesUntil('\r', writeBuffer, WRITE_BUFFER_SIZE - 1);
							writeBuffer[readBytesCount] = '\0';
							if (strncmp(writeBuffer, boundaryValue, strlen(boundaryValue)) != 0) {
								file.print(writeBuffer);
								while (ethernetClient.peek() == 10 || ethernetClient.peek() == 13) {
									file.print((char) ethernetClient.read());
								}
							} else {
								break;
							}
						}
						file.close();
						fileWritten = true;
					}
				}
			}
		}
	}
	return fileWritten;
}

It reads up to 128 bytes of a line of data at a time.
Checks to see if the read bytes starts with the boundary value.

If the data doesn't start with the boundary value then i write the data to file.
Then read any line ending characters that might now be at the start of the Stream and write these to file.
Repeat - read another 128 bytes.

If the data does start with the boundary value then i close the file - the file data is now written.

The method needs some tweaking - it needs to be compatible with linux as well as windows line endings.
But it works for now :slight_smile: .