Processing large POST requests from an ESP8266?

Hello. For my arduino project I want to make a web server with an ESP8266 (ESP-01) wifi module. And after that send a pretty large POST request. I'm talking about 0.5-1.5Mb in size. Then I want to process this request as it arrives to my arduino, byte by byte, without the buffer overflowing. The problem is that in the meantime, my arduino has to execute certain tasks based on the data that is being received, and these tasks can take from a few ms to multiple seconds to execute, so I can't just stand there and listen for the data coming in. So can my ESP8266 respect the serial buffer of my arduino and wait until there is enough free space, to send another part of the POST request?

No. You should structure your program so that no task takes longer than the time it takes to fill the buffer. You do one task, check the buffer, do another task, check the buffer.

Usually it's possible to write all tasks to fit into that time, so your loop does:

void loop() {
  doTasks();
  doSerialInput();
}

MorganS: No. You should structure your program so that no task takes longer than the time it takes to fill the buffer. You do one task, check the buffer, do another task, check the buffer.

Usually it's possible to write all tasks to fit into that time, so your loop does:

void loop() {
  doTasks();
  doSerialInput();
}

Yes, but the problem is that executing a task takes seconds. I can do parts of the task, then come back and do serial input, but the arduino just doesn't have enough ram to buffer all the incoming tasks. That's why I need to execute a task, get a new one from the POST request, execute it ....and go until there are no tasks left.

Yes, but the problem is that executing a task takes seconds.

The "help me with my hand waving" forum is down the road a ways.

Post CONCRETE examples of a task that takes several seconds. Use of delay() in that task does NOT count. You should NOT have a single delay() call in your code.

Why would you be POSTing large amounts of data to the Arduino? POST data that is small and easy to parse. Leave extracting data from novels for computers that have the memory and speed to do it.

KingOfAllChunks:
I’m talking about 0.5-1.5Mb in size.

I agree with @PaulS. To my mind it is crazy to send that much data to an Arduino. That is a job for a laptop or a RaspberryPi.

…R

can my ESP8266 respect the serial buffer of my arduino and wait until there is enough free space?

I believe the ESP8266 supports hardware handshaking... I think someone else had a similar problem, and they used another hardware pin to implement it (the UART does not provide this function).

Big questions remain, though: what could possibly take a few seconds, and what could an Arduino do with 500kB?

If I suspend my disbelief for a second, I would suggest hooking to the RX char interrupt. If you are just wading through the bytes, waiting for a few keywords or values, you could do that during the interrupt. Then you don't have to worry about getting back to the read() before the input buffer overflows. You would ignore most chars in the interrupt, saving the time it takes to put them in the buffer and take them back out again. I have posted a modified version of HardwareSerial, NeoHWSerial, that allows you to attach a function to be called with each received character.

But I'd like to see your sketch before claiming that's a good solution.

Cheers, /dev

PaulS: Post CONCRETE examples of a task that takes several seconds.

All right. How about moving two steppers? The incoming data represents the different steps that each motor has to take (a byte is a different offset for a motor, and the next byte is an offset for the other motor).

Robin2: it is crazy to send that much data to an Arduino.

What I can try to do is send bursts of data, about 32 or 48 bytes in size so that they don't fill up the serial buffer. But my objective is to send all the data at once and let the arguino execute the tasks after that. It's critical to not have any sudden short stop of the motors once they're moving (which can be avoided if all the data is already transmitted and the esp works with software/hardware handshaking). Otherwise if I send short packages of data, the arduino has to wait to receive the next package, which will result in a short stop of the motors.

/dev: But I'd like to see your sketch before claiming that's a good solution.

The sketch isn't a thing yet since I don't know a solution to the problem with sending all the data at once and letting the arguino read it byte by byte when needed. What should be in the sketch is a simple function for listening the data, then if there's data - parse it. From the parsing move the motors by x and y number of steps.

EDIT: How about reprogramming the ESP? Can I program it directly using arduino sketches?

KingOfAllChunks: All right. How about moving two steppers?

I have a Python program that sends data to an Uno to control 3 steppers.

It is quite impractical to send individual step signals from a PC to an Arduino and have them implemented at an accurate speed even using a USB connection. With WiFi it is even less practical. It used to be possible in the old days when PCs has a parallel printer port.

My code sends all the information for a complete move to the Arduino. That information is less than 64 bytes so it can be accommodated in the Serial Input buffer while the Arduino is moving the motors. When the move is finished the data for the next move is waiting and, before starting the move, the Arduino tells the PC to send more data.

...R

my objective is to send all the data at once and let the arduino execute the tasks after that… if I send short packages of data, the arduino has to wait to receive the next package, which will result in a short stop of the motors.

Like Robin2 is suggesting, you have to:

  • send higher-level “move” commands, not individual steps. This will decrease the amount of data that has to be sent. It’s very easy for the Arduino to convert endpoints to the required individual steps.
  • make the Arduino “ask” for more data, instead of trying to keep up with the PC. You may want both an ACK response (Arduino got the command ok), and a READY response (Arduino can take another command). Checksums would be a good idea, if you’re not already doing that.
  • Structure the sketch so it can keep doing little parts of the whole problem, continuously. Your loop should do something like (1) receive and parse one char of a command packet1, (2) see if it’s time to step the motor, (3) send a READY response, etc. and repeat. NO DELAYS! This is very different from a “batch” approach, where you have the entire job in memory and there’s no interaction with the PC once it starts.Developing for the embedded environment is very different, at first. The structure of an Arduino sketch is usually very different from a PC program, in that it never stops, must never stop.

since I don’t know a solution to the problem with sending all the data at once and letting the arguino read it byte by byte when needed.

Many Arduino questions can be traced back to this fundamental difference: trying to make it stop with a delay, or waiting until you’ve received all the characters is the wrong solution. “Blocking” like this is bad in the Arduino environment. So keep goin’, all the times!

Cheers,
/dev


1Robin2 has an example of Serial Input here, I have posted this for gradually parsing commands, and Nick Gammon has a great description of Finite-State machines here.

KingOfAllChunks: All right. How about moving two steppers? The incoming data represents the different steps that each motor has to take (a byte is a different offset for a motor, and the next byte is an offset for the other motor).

No, you just have to store the commands in a buffer and carry on. The separate task of interpretting those commands and sending pulses to the stepper motor would be regularly progressed by a call from loop.

Have you studied the BlinkWithoutDelay example?

All right. How about moving two steppers? The incoming data represents the different steps that each motor has to take (a byte is a different offset for a motor, and the next byte is an offset for the other motor).

If you have any control over the data being sent, then you need to send the data in a format that is easy to parse. Below is simple servo command code that parses the desired servo, and the value to be sent to that servo.

//zoomkat 11-22-12 simple delimited ',' string parse 
//from serial port input (via serial monitor)
//and print result out serial port
//multi servos added 
// Powering a servo from the arduino usually *DOES NOT WORK*.

String readString;
#include <Servo.h> 
Servo myservoa, myservob, myservoc, myservod;  // create servo object to control a servo 

void setup() {
  Serial.begin(9600);

  //myservoa.writeMicroseconds(1500); //set initial servo position if desired

  myservoa.attach(6);  //the pin for the servoa control
  myservob.attach(7);  //the pin for the servob control
  myservoc.attach(8);  //the pin for the servoc control
  myservod.attach(9);  //the pin for the servod control 
  Serial.println("multi-servo-delimit-test-dual-input-11-22-12"); // so I can keep track of what is loaded
}

void loop() {

  //expect single strings like 700a, or 1500c, or 2000d,
  //or like 30c, or 90a, or 180d,
  //or combined like 30c,180b,70a,120d,

  if (Serial.available())  {
    char c = Serial.read();  //gets one byte from serial buffer
    if (c == ',') {
      if (readString.length() >1) {
        Serial.println(readString); //prints string to serial port out

        int n = readString.toInt();  //convert readString into a number

        // auto select appropriate value, copied from someone elses code.
        if(n >= 500)
        {
          Serial.print("writing Microseconds: ");
          Serial.println(n);
          if(readString.indexOf('a') >0) myservoa.writeMicroseconds(n);
          if(readString.indexOf('b') >0) myservob.writeMicroseconds(n);
          if(readString.indexOf('c') >0) myservoc.writeMicroseconds(n);
          if(readString.indexOf('d') >0) myservod.writeMicroseconds(n);
        }
        else
        {   
          Serial.print("writing Angle: ");
          Serial.println(n);
          if(readString.indexOf('a') >0) myservoa.write(n);
          if(readString.indexOf('b') >0) myservob.write(n);
          if(readString.indexOf('c') >0) myservoc.write(n);
          if(readString.indexOf('d') >0) myservod.write(n);
        }
         readString=""; //clears variable for new input
      }
    }  
    else {     
      readString += c; //makes the string readString
    }
  }
}

<zoomkat’s program>

Do not use String. Char arrays are fine, but the String class is a terrible choice, and will eventually cause programs to fail in a bewildering number of ways. From a previous discussion:

…we have zoomkat’s abominable String implementation. Not only does it add 1.6k to the program size, use dynamic memory in a limited-RAM environment, and buffer the entire line (i.e., handle each character multiple times), it searches for a character that might be present, multiple times. “Searching” is a dirty word when it comes to parsing.

Shame on you for suggesting something that may work now, but is inefficient, wasteful, and the eventual cause of endless headaches later.

Cheers,
/dev

Shame on you for suggesting something that may work now, but is inefficient, wasteful, and the eventual cause of endless headaches later.

Did your corn flakes taste like pee again? You have your opinion, so post your code.

KingOfAllChunks:
Hello. For my arduino project I want to make a web server with an ESP8266 (ESP-01) wifi module. And after that send a pretty large POST request. I’m talking about 0.5-1.5Mb in size. Then I want to process this request as it arrives to my arduino, byte by byte, without the buffer overflowing.

I have a library that does exactly that. Gammon Forum : Electronics : Microprocessors : Tiny web server for Arduino or similar

It processes incoming text “on the fly”. You only need enough RAM to hold a single element (key and value) up to a maximum size you specify (eg. 100 bytes). You can also do other things while the incoming text is arriving. When a key/value pair completely arrives a user-specified callback function is called, which then handles that particular key/value pair.

The below TextFinder application might be of interest.

http://playground.arduino.cc/Code/TextFinder

zoomkat: The below TextFinder application might be of interest.

A response from the OP would be even more interesting.

...R