Parsing HTTP request string manually

Projects Networking, Protocols, and Devices
1

I need to call an HTTP request parser with a string that contains the request data and have it return an object with parsed results. I know that if I actually set up a WebServer instance with WiFi, this is all automatic but I'm not actually getting the request from a web server, but from a serial port. Thanks.

Here is a long story that may not be relevant:

I have done some searches and what I got were all very old posts. I have not used Arduino wifi or http for a very long time. Did my fair share around 2012 and later I used ESP32 and ESP-IDF. Now I'm faced with this issue that I think the solution is an HTTP request parser.

I have an IR remote sample code from IR remote ESP8266 library that I updated to handle some specific HTTP requests. The code uses the following libraries:

WebServer.h
WiFi.h
IRremoteESP8266.h

So the code is really easy to update based on the WebServer callback functions attached to each URI and use server.arg(i), server.argName(i), and server.args() to extract each parsed argument. I use these to construct a map (dictionary) to search for relevant parameters.

This is all good but I need this program integrated into a bigger system where the code doesn't receive HTTP request but rather receives requests via a serial port. So I need to receive requests in the form of a string and call an HTTP request parser to parse the string.

2

I think I found where WebServer library does the parsing:

void WebServer::_parseArguments(String data)

Only problem is, this is mixed into the WebServer library and not an easy stand-alone function I can feed it string and expect some parsed structs.

My guess is, once Arduino has this WebServer library, the stand-alone library efforts to parse HTTP requests eventually stopped.

Is there a stand-alone parser library that's relatively recent? Thanks.

3

What’s the format of your request?

The typical format for a query string URL is:

scheme://host/path?key1=value1&key2=value2&…

It starts with the URL’s scheme (http or https), followed by the host and optional path.

The ? introduces the query string, which contains one or more key=value pairs separated by &.

Both keys and values should be URL-encoded if they contain special characters.

➜ does this match what you have and do you have encoding?

4

Thanks for responding! Yes, it's an HTTP GET or HTTP POST request. I wonder if there's still active libraries for GET. Meanwhile, I've encountered arduinojson 7 library. I CAN make the main processor send json to the arduino so this is likely a more desirable format than the old http GET format.

5

I don't think it matters much if it is from a http request or a string that you got from a serial port. If it is a JSON string you could use a JSON library to parse command to parse it. If it isn't JSON could you just use functions from the string.h library or String library to parse it.

6

An HTTP server does not deal with a whole URL. The origin -- scheme, host, and port -- has already be used to contact (one of) the server(s) there. The first word in an HTTP request is the method/verb like GET or POST; then the rest of the URL, starting with the slash in front of the path; e.g.

GET /some/path?a=1&b=2 HTTP/1.1

After that first request-line, there can be several request headers and sometimes a request body, including for when the request comes from a form with a POST. Using a from with a GET, the form values are sent as query parameters in the request-line instead.

So that's a big difference. Do you need to handle both? You may be better off with JSON, since it also expresses whether a value is a number versus a string (or a boolean).

7

I think we are drifting further from my OP, which was to find an HTTP GET request parser library that is recent. I didn't ask about what the headers are, just the request part of the GET address. Then someone mixed up my response as "he also wants to parse a JSON from HTTP". No, please read and respond, not just respond.

8

So here's an out of left-field suggestion.

If a webserver can parse what you want, then why not present the data to a webserver?

  • Open webserver on localhost
  • Read data from serial port
  • Open an HTTP client on localhost
  • Send the data from client to server as a GET request
  • Server parses serial data and provides it in a callback

In theory it should work and it might be quicker than stripping out the parsing code from an existing server.

9

Thanks. That's an interesting solution and could work nicely. I'll give it a try!

10

Exactly

That’s why before jumping to conclusions I wanted to clarify what was received on the Serial port.

When a client makes a request to a URL, the scheme indicates the protocol and default port, the host is resolved to an IP address by the networking stack using DNS, the port is selected either from the scheme or explicitly from the URL, and once the client establishes a TCP or TLS connection to that host and port, the HTTP server code takes over to process the incoming request.

So if on the client what was asked is to request

http://host/path?key1=value1&key2=value2

Then what the http server gets is a well formed GET request, may be something like

GET /path?key1=value1&key2=value2 HTTP/1.1
Host: host
User-Agent: SomeClient/1.0
Accept: */*

The first line is the request line with method, path including query string, and HTTP version. Then come the headers, one per line. An empty line follows to mark the end of headers and no body is sent with a GET request.

So what I was trying to assert was if OP gets on the Serial line the former or latter…

It’s not the same parsing.