Is this the best way to GET data from a HTTP request?

I'm an old web programmer. I spent many years writing PERL and PHP and eventually I got burned out. But on rare occasion, I get an idea that I want to try and dust off the old code writing tools. My c++ game is weak, I admit.

I need to parse a GET string into useful data. I've come to terms with the fact that I'm limited to single character parameter names in order to make the code efficient and avoid the String class. That's fine. I'm only looking to handle a half dozen parameters at most.
But the value can not be limited to a single integer. The value may be 2 it may be 2000 or it may be 0.002 (not implemented in the code, but definitely feasible)

After several days of failure, frustration and useless google results (every other attempt I've read uses String), I think I have something that works. But I'm not convinced it's the best way.

The end goal is to pass data via WiFi to the Arduino via HTTP requests and have it send a response back as JavaScript code, most likely altered variables at which point, the browser would execute a new action based on the response.

Pretend we have a bi-ped robot.

Sample HTTP request: "GET /action?s=45&d=90 HTTP/1.1 (more lines for host etc.)". Hypothetical action "turn 90 degrees, take 45 steps and wait for the next command. It would be up to the browser to determine that if you should walk 45 steps then turn, the command would be "...s=45&d=0" followed by "s=0&d=90" Or maybe the Arduino would walk in an arc taking one step and then turning 2 degrees before the next step for the combined request... I haven't thought that far into this example. Throughout the walk, the browser would check in on the arduino to see if it's done yet. This request would be "GET /info?o=1 HTTP/1.1". Every few seconds, it asks "are you done yet?" and the arduino would respond accordingly. (currently returning the arduino clock for example purposes)

What I'd like to know is iterating over the GET request one character at a time and acting on each character the best way to handle a string of data without using a String?

Secondly, is there any way to work with the 80-character GET_str that doesn't require the code to make sure it's not simply a shorter string than 80 characters and stop the iteration in a for loop? Is a

while (i<sizeof(x) && x[i] != NULL){
...
i++;
}

the only other option?

#include <ESP8266WiFi.h>

const char* ssid = "Verizon";
const char* password = "12345678";

// Create an instance of the server
// specify the port to listen on as an argument
WiFiServer server(80);

void setup() {
  pinMode(LED_BUILTIN, OUTPUT); //set the built in LED to an output
  digitalWrite(LED_BUILTIN, 1);
  Serial.begin(115200);
  delay(1000);  
  
  // Connect to WiFi network
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  WiFi.begin(ssid, password);
  
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  digitalWrite(LED_BUILTIN, 0); //turn the LED on to show connected to WiFi
  
  // Start the server
  server.begin();
  Serial.println("Server started");

  // Print the IP address
  Serial.println(WiFi.localIP());
}

void loop() {
  // Check if a client has connected
  WiFiClient client = server.available();
  if (!client) {
    return;
  }
  
  // Wait until the client sends some data
  Serial.println("new client");
  while(!client.available()){
    delay(1);
  }
  
  client.flush();
  
  // Read the first line of the request
  char c;
  int i = 0;
  int spaces_received = 0;
  int bufindex = 0; // reset buffer
  char GET_str[80] = {};
  char store_value;
  int steps = 0;
  int degree = 0;
  int question_mark_position = 0;

  // read the first line to determinate the request page
  while (c != '\r') {
    c = client.read();

    if(c == ' '){
      spaces_received++;
      continue;
    }
    if(spaces_received == 0) continue; //HTTP request starts with a method followed by a space, so until we get the first space, ignore it.
    if(spaces_received > 1) break; //if we already got a seconds space, we have all the information we need, stop collecting it.

    GET_str[bufindex] = c;
    /*Serial.print(bufindex);
    Serial.print(':');
    Serial.println(c);
    */
    bufindex++;
  }
  if(GET_str[1] == 'a'){ //perform the action routine url: /action/x?foo=bar&et=al
    for (i=0; i<sizeof(GET_str); i++) {
        if(GET_str[i] == '?'){
          question_mark_position = i;
          //Serial.print("Question Mark Position:");
          //Serial.println(i);
          continue; //now we know the start position of the get string; No need to store it.
        }
        if(question_mark_position == NULL) continue;
        if(GET_str[i] == '&'){ //new variable to store, reset the store_value and skip this iteration
          Serial.println("Found an ampersand");
          store_value = NULL;
          continue;
        }
        if(GET_str[i] == NULL) continue; //most likely the request already ended, but we're going to keep looping to be sure, just skipping for now to prevent arithmetic problems.
        if(GET_str[i] == 'd' || store_value == 'd'){
          store_value = 'd';
          if(GET_str[i] == 'd' || GET_str[i] == '=') continue; //don't store the variable name or equals sign.
          Serial.print("storing d. ");
          Serial.print(" old value: ");
          Serial.print(degree);
          degree *= 10; //move the next numerically stored one slot to the left;
          Serial.print(" temp value: ");
          Serial.print(degree);
          Serial.print(" adding ");
          Serial.print(GET_str[i]);
          
          degree = degree + ((int) GET_str[i] - 48); //the interface received an ASCII character, which is 48 decimal higher
          Serial.print(" new value ");
          Serial.println(degree);
        }else if(GET_str[i] == 's' || store_value == 's'){
          store_value = 's';
          if(GET_str[i] == 's' || GET_str[i] == '=') continue; //don't store the variable name or equals sign.
          steps *= 10; //move the currently numerically stored decimal one slot to the left;
          steps = steps + ((int) GET_str[i] - 48); //the interface received an ASCII character, which is 48 decimal higher
        }
    }
  }else if(GET_str[1] == 'i'){ //perform the info routine currently outputs arduino clock
    client.print("HTTP/1.1 200 OK\r\nContent-Type: text/javascript\r\n\r\n"); //HTTP header
    client.print("var current_time = \"");
    client.print(millis());
    client.print("\";");
    return;
  }else{ // everything else is a 404
    client.print("HTTP/1.1 404 NOT FOUND\r\nContent-Type: text/html\r\n\r\n"); //HTTP header
    client.print("The file you are looking for ");
    for (i=0; i<sizeof(GET_str); i++)
      if(GET_str[i] != NULL)
        client.print(GET_str[i]);
    client.print(" could not be found.");
    return;
  }
  
  client.print("HTTP/1.1 200 OK\r\nContent-Type: text/javascript\r\n\r\n"); //HTTP header
  client.print("var current_time = \"");
  client.print(millis());
  client.println("\";");
  client.print("var degree = \"");
  client.print(degree);
  client.println("\";");
  client.print("var steps = \"");
  client.print(steps);
  client.println("\";");
  client.flush();

  delay(1);
  Serial.println("Client disonnected");

  // The client will actually be disconnected 
  // when the function returns and 'client' object is detroyed
}

I suspect the code runs directly on the ESP8266? Stop worrying about strings. People used to have PCs that where slower and run word processing on them. :slight_smile:

If this is a module connected to a small Arduino just get yourself an Arduino with an ARM Cortex-M processor. Some of them are even cheaper then classic Arduinos and they will all eat strings for breakfast.

The best code can be the one that is easy to understand, can be modified and maintained by others and not necessarily the one using the least amount of hardware resources.

I don't disagree. I often have said "Good code is self documenting" to encourage my team to use very sensible variable and function names and think before coding. Yes, this is a knock-off WeMos D1 R1 ESP8266. It's biggest weakness is the complete lack of documentation. I posted a comment on Amazon with the URLs for the driver, the board manager link and a URL with a drag and drop code block example, sadly, Amazon removed it. It also only has one analog port, which right now, I'm probably not going to use any analog pins anyway.

Since I've taken the time to figure this out, using char arrays, is there something that would be better? Feel free to comment on my coding style, it's been a while since I've had anyone comment on my code.

if you are on an ESP and if you want to write a webserver please follow the Example

ESP8266Webserver | HelloServer

how to build a webserver on the ESP. Let the ESP8266Webserver library write the HTTP header fields. Use the nice methods to read the parameters (.args, .arg)!

First, there is NO problem at all using String on the ESP. It has more than plenty of memory to deal with it unless you're being very reckless.

Second, if you still don't want to use String, get familiar with strtok. It will simply and easily parse the GET arguments into a series of tokens, without having to do any additional memory allocation.

Regards,
Ray L.

smacaroni:
Feel free to comment on my coding style, it's been a while since I've had anyone comment on my code.

Your code looks clearly better than many posts. Since you asked here a few things I might do different (not necessarily better).

  • your code creates a few compiler warnings which I would try to remove
  • for Arduino code I use the recommended naming convention for variables (camelCase)
  • use HIGH and LOW in digitalWrite instead of 0 and 1 (same reason staying with convention)
  • I try to remove any comments that are obvious unless it is in a place where I expect others to do modifications
    e.g. this comment // Start the server and this one // Print the IP address
  • I try to avoid while loops
  • and my goal is to have loop() run continuously and fast, so I can add new functionality
  • like most people helping on the forum I avoid the delay() function and use the millis timer for time control
  • I declare loop variables in the loop, that tells the compiler two loops are independent
  • I use #define for the LEDs and give it a functional name e.g. #define WIFI_LED_PIN LED_BUILTIN

Thank you for taking the time to examine the code. There is some code that is left from the script I used as inspiration, which is most of the web server code. I'd really like to find the original code again if anyone recognizes it. A lot of comments were for the purpose of this hypothetical, I'm pretty sure had I left all the comments out, the majority of the code would be self explanatory.

But I am curious, where can I make the loop run faster? This would definitely help me in my end goal.

smacaroni:
But I am curious, where can I make the loop run faster? This would definitely help me in my end goal.

I use a function that handles all WiFi stuff (setup, reconnects, communication) and nothing else and call that a task. The task uses functions to keep the complexity down. But these functions are not used anywhere else unless they are generic. I use a state machine based on switch-case and run only a little bit of the code at any one time. The tasks never stop (delay) and does not wait for anything (while loops).

Every time the code runs through loop the WiFi task is called and then other tasks. The tasks communicate over very few global variables and flags and the state is kept in local static variables.

This way every task gets a time slice every time loop() runs. As the code is expanded, I monitor the number of loops per second and the minimum and maximum loop time via the Serial Monitor. It is a simple performance analysis. :slight_smile:

If I find that a certain part of the code takes longer than a few cycles and cannot be split, I make sure it is only called as often as necessary by using millis().

Thanks. I definitely need to write a re-connect routine as I have a tendency to walk away with my phone to another part of the house and then wonder why my stuff suddenly won't work when I come back.
I also avoid the delay(), that's another artifact from the base WiFi HTTP server code I started with.

I have a strtok() like function for breaking up GET parameters (A.K.A. the query string). It relies on the query string being writable, and moves along the string replacing separators with '\0' ( the string terminator), then passing back the two strings for a particular query pair, plus a new start position for the next time through the loop. Since it's all done in place, there's no memory overhead (but the query string is mangled). It's part of a simple web-server I wrote.

This is copied from that code I came up with (so not tested as is, but should be close):

	//  Declarations
	struct queryPair
	{
		const char *_name;
		const char *_value;
	};

	//  Breaks apart a query string into name / value pairs.  Initially, call with the
	//  original query string.  Use the return value on subsequent invokations to move
	//  along the query string.  You're done when it return 'nullptr'.
	//
	//  The 'queryPair' structure will have pointers to both the name and the value
	//  components.  Each will be already URL decoded.
	//  - if the value is 'nullptr', then the original query string didn't have a value
	//    for that name.
	//  - the value can also be a zero length string.
	char *getNextQueryPair(char *queryString, queryPair &nameValuePair);

	//  Decodes a URL encoded string in place.  A decoded string is *never* longer than
	//  the original string, so we don't need to know how long the buffer is.
	void urlDecode(char *encodedString);


	//  Implementation

	// URl decoder, taken from
	// https://stackoverflow.com/questions/2673207/c-c-url-decode-library/19826808
	// License is CC-BY-SA https://creativecommons.org/licenses/by-sa/4.0/
	void urldecode2(const char *src, char *dst)
	{
		if (!src || !dst) return;

		char a, b;
		while (*src)
		{
			if ((*src == '%') &&
				((a = src[1]) && (b = src[2])) &&
				(isxdigit(a) && isxdigit(b)))
			{
				if (a >= 'a')
					a -= 'a' - 'A';
				if (a >= 'A')
					a -= ('A' - 10);
				else
					a -= '0';
				if (b >= 'a')
					b -= 'a' - 'A';
				if (b >= 'A')
					b -= ('A' - 10);
				else
					b -= '0';
				*dst++ = 16 * a + b;
				src += 3;
			}
			else if (*src == '+')
			{
				*dst++ = ' ';
				src++;
			}
			else
			{
				*dst++ = *src++;
			}
		}
		*dst++ = '\0';
	}

	//  In place URL decoder.  Decoding *never* produces a string larger than its input.
	void urldecode(char *srcdst)
	{
		return urldecode2(srcdst, srcdst);
	}

char *getNextQueryPair(
	char *queryString,
	queryPair &nameValuePair)
{
	nameValuePair._name = nullptr;
	nameValuePair._value = nullptr;

	if ((queryString == nullptr) || (*queryString == '\0'))
	{
		return nullptr;
	}

	char *retVal = nullptr;

	char *end = strchr(queryString, '&');

	if (end != nullptr)
	{
		*end = '\0';
		retVal = end + 1;
	}

	char *split = strchr(queryString, '=');

	if (split != nullptr)
	{
		*split = '\0';
		split++;
	}

	//  Now that things have been split apart at the markers, we can finally decode the
	//  strings.
	urlDecode(queryString);
	urlDecode(split);

	nameValuePair._name = queryString;
	nameValuePair._value = split;

	return retVal;
}



//  Usage

void ProcessQueryString(char *pQueryString)
{
   queryPair nameValuePair;

   while (pQueryString != nullptr)
   {
       pQueryString = getNextQueryPair(pQueryString, nameValuePair);

       //  nameValuePair._name and nameValuePair._value are the split query pair.
       
       //  Process your stuff here
   }

}

This is all from the library YAAWS that I put together.

as the thread owner is on an ESP8266 , parameter read can be done by these simple methods:

server.args()
server.argName()
server.arg()
server.hasArg()

noiasca:
as the thread owner is on an ESP8266 , parameter read can be done by these simple methods:

server.args()
server.argName()
server.arg()
server.hasArg()

I'm certainly going to look into this more, found a decent tutorial at ESP8266 Webserver: Getting query parameters - techtutorialsx hopefully this all ties together.

Before I saw this, I tried something else which is giving me really strange results. Normally, I would just go with the above tip and explore that, but clearly there's something massively wrong in my very limited understanding of c++ that needs addressed.
I moved the if() block if(GET_str == 'd' || store_value == 'd')[/color]
into a function, two functions actually and I don't understand why it's not printing to the serial in the function.
```
*int store_from_get(char str_array[80], int i){

int tmp = 0;

int store_value_sign = 1;

for(int i=i;i<sizeof(str_array);i++){

if(str_array[i] == '&') break; //if we found the next parameter, stop.

Serial.print("storing ");

Serial.print(" old value: ");

Serial.print(tmp);

tmp *= 10; //move the next numerically stored one slot to the left;

Serial.print(" temp value: ");

Serial.print(tmp);

Serial.print(" adding ");

Serial.print(str_array[i]);

if(store_value_sign == -1){

tmp = tmp - ((int) str_array[i] - 48); //the interface received an ASCII character, which is 48 decimal higher

}else{

tmp = tmp + ((int) str_array[i] - 48); //the interface received an ASCII character, which is 48 decimal higher

}

Serial.print(" new value ");

Serial.println(tmp);

}

return tmp;

}

int numdigits(int i){

char str[20];

sprintf(str,"%d",i);

return(strlen(str));

}*
* *Which is called by:* *
if(GET_str[i] == 'd'){
    Serial.println("start store_from_get");
    i += 2; //skip the param name and the equals sign.
    degrees= store_from_get(GET_str,i);
    i += numdigits(degrees); //slide over in the GET string the width of the value to reduce the number of loops.
    Serial.println("finished store_from_get");
}

* *The issue is store_from_get returns 0 and Serial.print doesn't echo anything so I can't debug the function if the Serial doesn't print.* *I have tried adding Serial.begin(115200);* *inside the function. I have tried sending the value of the i for the for loop to p and then setting i to the value of p.* *And most strange of all, this is the serial monitor output, it completely drops the "starting" line and replaces it with Fo and a backwards question mark?* *
new client
Fo⸮finished store_from_get
Found an ampersand
Client disonnected

```
So what's gone haywire in my function? Does Serial.print not work inside functions?

Well, I didn't get the no-Strings working the way I wanted, but I now have a half decent query string interface which does the other part of what I set out to do. The server.arg*() is dependent upon the Strings as far as I can tell.
This example is more simplistic, it turns the LED on for a period of time with an optional delay and responds to two different paths.

Action path turn on the LED for a given period of time with an optional delay.
http:// 127.0.0.1/action?s=(integer in milliseconds to stay on)&d=(integer in milliseconds to wait before turning on)

Hello World
Current_time = 441242
turn on time: 456242
turn off time: 458242
Arg nº 0 -> s: 2000
Arg nº 1 -> d: 15000

Info path return the state of the system in the confines of the LED turning on/off routine.
http://192.168.1.6/info

Three possible results.
LED Off

Hello World
Current_time = 495306
LED State = Off
turn on time: NULL
turn off time: NULL

LED to turn on at a set period of time

Hello World
Current_time = 551629
LED State = Off
Turn on in: 6806
turn on time: 558435
turn off time: 560435

LED on waiting to turn off

Hello World
Current_time = 602625
LED State = On
On Time Remaining: 129
turn on time: 600754
turn off time: 602754

Code (everything but the LED bits and the Action/Info handler are from the tutorial listed above)

#include "ESP8266WiFi.h"
#include "ESP8266WebServer.h"
 
ESP8266WebServer server(80);
int turn_on_time = NULL;
int turn_off_time = NULL;
 
void setup() {
 
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT); //set the built in LED to an output
  WiFi.begin("HelpLetMeOutOfHere", "12345678");  //Connect to the WiFi network

 
  Serial.println("Waiting to connect…");
  while (WiFi.status() != WL_CONNECTED) {  //Wait for connection
    delay(500);
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); //simple method to flash the LED
  }
 
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());  //Print the local IP
 
  server.on("/action", handleActionPath);    //Associate the handler function to the path
  server.on("/info", handleInfoPath);    //Associate the handler function to the path
  server.begin();                    //Start the server
  Serial.println("Server listening");
 
}
 
void loop() {
  server.handleClient();         //Handling of incoming requests
  if(turn_on_time == NULL)
    digitalWrite(LED_BUILTIN, 1);
    
  if(turn_on_time != NULL && turn_on_time < millis())
    digitalWrite(LED_BUILTIN, 0);
    
  if(turn_off_time != NULL && turn_off_time < millis()){
    digitalWrite(LED_BUILTIN, 1);
    turn_on_time = NULL;
    turn_off_time = NULL;
  }
}
 
void handleActionPath() {
  int current_time = millis();
  if(server.hasArg("s")){
    turn_on_time = current_time;
    turn_off_time += turn_on_time + (int) server.arg("s").toInt();
  }
  if(server.hasArg("d") && server.arg("d").toInt() > 0){
    turn_on_time += (int) server.arg("d").toInt();
    turn_off_time += (int) server.arg("d").toInt();
  }
  Serial.print("Sending Reply");
  String message = "<html>\n<head>\n<body>\nHello World\n<pre>\n";
  message += "Current_time = ";
  message += millis();
  message += "\nturn on time: ";
  message += (turn_on_time == NULL) ? "NULL" : (String) turn_on_time;
  message += "\nturn off time: ";
  message += (turn_off_time == NULL) ? "NULL" : (String) turn_off_time;
  for (int i = 0; i < server.args(); i++) {
    message += "\nArg nº " + (String) i + " -&gt; ";
    message += server.argName(i) + ": ";
    message += server.arg(i);
  }
  message += "</pre>\n</body>\n</html>";
  server.send(200, "text/html", message);
 
}

void handleInfoPath() {
  Serial.print("Sending Reply");
  String message = "<html>\n<head>\n<body>\nHello World\n<pre>\n";
  message += "Current_time = ";
  message += millis();
  message += "\nLED State = ";
  message += (digitalRead(LED_BUILTIN) == 1) ? "Off" : "On";
  if(digitalRead(LED_BUILTIN) == 0){
    message += "\nOn Time Remaining: ";
    message += turn_off_time - millis();
  }
  if(turn_on_time != NULL && digitalRead(LED_BUILTIN) == 1){
    message += "\nTurn on in: ";
    message += turn_on_time - millis();
  }
  message += "\nturn on time: ";
  message += (turn_on_time == NULL) ? "NULL" : (String) turn_on_time;
  message += "\nturn off time: ";
  message += (turn_off_time == NULL) ? "NULL" : (String) turn_off_time;
  for (int i = 0; i < server.args(); i++) {
    message += "\nArg nº " + (String) i + " -&gt; ";
    message += server.argName(i) + ": ";
    message += server.arg(i);
  }
  message += "</pre>\n</body>\n</html>";
  server.send(200, "text/html", message);
}

if you are trying to use an API to read something in JSON, it's real easy.

Worked for me the first time, that kind of easy. I got the stuff from the site I wanted into the serial monitor copied it to Assistant | ArduinoJson 6 which then gives all the code for your arduino to search what you want and put it in variables you can read.

Piece of cake.

sevenoutpinball:
if you are trying to use an API to read something in JSON, it's real easy.

Worked for me the first time, that kind of easy. I got the stuff from the site I wanted into the serial monitor copied it to Assistant | ArduinoJson 6 which then gives all the code for your arduino to search what you want and put it in variables you can read.

Piece of cake.

The response is easy part. In fact, I don't even need to get into JSON. I'm simply going to return new values for previously defined variables as text/javascript

Current_time = "602625";
LED_state = "1";
turn_on_time = "600754";
turn_off_time = "602754";

The reason everything is in quotes is it'll prevent JS errors that might break the script if something non-numeric is returned, using parseFloat or parseInt and isNaN will make sure that any garbage data can elegantly handed (such as calculating "on time remaining" or "turn on in", when a NaN is returned) The script can output an error message or not, but most important it won't break the code. You do need to be a little more careful with null strings though. null is valid "null" is a string creating a case where you have a NaN situation which is probably not what one would want.

I.E.

String message = "turn_on_time = ";
message += (turn_on_time == NULL) ? "null" : "\"" + (String) turn_on_time + "\"";
message = ";\n";

The above certainly could be serialized, but it's quite a bit of additional work for four variables. If I had 60, I definitely would.