Arduino Uno interface ESP8266 POST request EC2 webserver - Bad Request

Hi, I have started arduino programming. This is my first mini project where I am trying to push some data on EC2 instance. I have a WSGI ( flask application ) running on EC2 which accepts GET and POST request. Using POSTMAN I am easily able to get and push data on the server.

For instance:
GET request: http://ipaddress:80/getdata
I get the json response from the server and returns 200

POST request: http://ipaddress:80/pushdata
I am able to push data and see it on the webserver.

However I am trying to use AT commands to POST the request and not able to solve the issue. I am able to connect to internet and also able to reach the host.

First Step: To able able to post the data from serial monitor to server:

AT + RST --> sucess

AT + CIPMODE=0 --> Sucess

AT + CIPSTART="TCP","ipaddress",80 --> Sucess

AT + CIPSEND=119

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:14\r\n{"hi":"hello"}

I do not get error on the serial terminal but I see Bad request 400 error on the server.

When I do the GET request I do not get any error on the server

GET /getdata HTTP/1.1

I have tried various variations of POST commands by changing different values but not able to post .

Second Step : To be able to use the same command in code to post the request

My aim is to post the data to the server through code using the ESP8266 module. However I got stuck in the first step only.

Any help is appreciated. I will be eventually post json formated data to to server from the code.

If this does not seem to work I am exploring sending data using MQTT to AWS IOT using ESP8266 . If anyone can help me in either case it would be much appreciated.

ESP 8266 Arduino
RX 11
TX 10
GND GND
VCC 5v
CH_RD 5v

Library I would be using to POST command in code:
#include<SoftwareSerial.h>
SoftwareSerial esp8266(RX,TX) //10,11

Thanks

a) I'm no fan of that Arduino-serial-ESP implementation. But in General a POST should send the content in the message body, where the meassage body is delimited from the html header by one line.

so instead of:

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:14\r\n{"hi":"hello"}

something like

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:14\r\n\r\n{"hi":"hello"}

might help, ...

b) change your implementation and programm the ESP.

Thanks for replying. I see what you are suggesting that the delimiter would try to distinguish between payload and header.
I think that is the exact problem I have. I tried below commads:

  1. added the delimiter as you suggested

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:14\r\n\r\n{"hi":"hello"}

on the server console i get exactly same command ss above with 400 bad request ( so yes it is not able to distinguish between payload and header)

  1. I tried posting without payload:

POST /pushdata HTTP/1.1

for which I get success message on server but the payload is null which is expected.

I do not know what more changes can help. It seems just a basic command. yet unable to fix it.

next thing I woud try would be a stepweise approach

POST /pushdata HTTP/1.1

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:0

POST /pushdata HTTP/1.1\r\nHost: xx.xxx.xx.xxx:80\r\nContent-Type: application/json\r\nContent-Length:14\r\n\r\n{"hi":"hello"}

there must be a reason why the resource doesn't accept your request.

additionally try to post manually with postman or something similar, trace the traffic with wireshark (if successfull) and compare it to the esp traffic.

I am able to get the data on the server with serial monitor command:

AT

OK

AT+CWJAP="PRATIK","pwd"
WIFI DISCONNECTED
WIFI CONNECTED
WIFI GOT IP

OK

AT+CIPSTART="TCP","xx.xxx.xx.xxx",80
CONNECT

OK
AT+CIPSEND=33

OK

GET /json/{"hi":"15432"} HTTP/1.1

busy s....

Recv 33 bytes

SEND OK

AT+CIPCLOSE

P.S : I connect rx of ESP8266 to rx of arduino ( pin 0) and tx of ESP8266 to tx of arduino (pin 1)

For the sequence of the above command in serial monitor at 115200 baud am successfully able to get the data which is {"hi","15432"} on the server. So now I have the sequence of command that works from serial monitor to send the data on the server.

I tried replicating the same in the code which does not work. Please find the code below:

I now connect rx of ESP8266 to 11 and tx of ESP8266 to 10 and run below code:

#include <SoftwareSerial.h>
#define RX 10
#define TX 11
SoftwareSerial esp8266(RX,TX); 

long timer = 0;
String AP = "PRATIK";      
String PASS = "pwd"; 
String HOST = "xx.xxx.xx.xxx";
String PORT = "80";

int countTrueCommand;
int countTimeCommand; 
boolean found = false; 
int valSensor = 1;

void setup() {
  Serial.begin(9600);
  esp8266.begin(115200);

  // test start up
  sendCommand("AT",5,(char *)"OK");

  sendCommand("AT+RST",5,(char *)"OK");
  // Connect to AP using username and pwd
  sendCommand("AT+CWJAP=\""+ AP +"\",\""+ PASS +"\"",20,(char *)"OK");
}
void loop() {
 String valSensor = "{\"HI\":\"HELLO\"}";
 String http = "HTTP/1.1";
 String getData = "GET /json/"+valSensor+" "+http;
 sendCommand("AT+CIPSTART=\"TCP\",\""+ HOST +"\","+ PORT,15,(char *)"OK");
  Serial.println(getData);
 sendCommand("AT+CIPSEND=" +String(getData.length()),4,(char *)">");
 esp8266.println(getData);
 delay(5000);
 countTrueCommand++;
 // close connections
 sendCommand("AT+CIPCLOSE",5,(char *)"OK");
}
void sendCommand(String command, int maxTime, char readReplay[]) {
  Serial.print(countTrueCommand);
  Serial.print(". at command => ");
  Serial.print(command);
  Serial.print(" ");
  while(countTimeCommand < (maxTime*1))
  {
    esp8266.println(command);//at+cipsend
    if(esp8266.find(readReplay))//ok
    {
      found = true;
      break;
    }

    countTimeCommand++;
  }

  if(found == true)
  {
    Serial.println("Sucess");
    countTrueCommand++;
    countTimeCommand = 0;
  }

  if(found == false)
  {
    Serial.println("Fail");
    countTrueCommand = 0;
    countTimeCommand = 0;
  }

  found = false;
 }

The output on the serial monitor is:
0. at command => AT Sucess

  1. at command => AT+RST Sucess
  2. at command => AT+CWJAP="PRATIK","hardsal2017" Sucess
  3. at command => AT+CIPSTART="TCP","xx.xxx.xx.xxx",80 Sucess
    GET /json/{"HI":"HELLO"}
  4. at command => AT+CIPSEND=39 Sucess
  5. at command => AT+CIPCLOSE Sucess
  6. at command => AT+CIPSTART="TCP","xx.xxx.xx.xxx",80 Sucess
    GET /json/{"HI":"HELLO"}
  7. at command => AT+CIPSEND=39 Sucess
  8. at command => AT+CIPCLOSE Fail
  9. at command => AT+CIPSTART="TCP","xx.xxx.xx.xxx",80 Sucess
    GET /json/{"HI":"HELLO"}
  10. at command => AT+CIPSEND=39 Sucess
  11. at command => AT+CIPCLOSE Fail
    ...
    ...
    ....
    ....

Output on the server is:

xxx.xxx.xxx - - [21/Sep/2018 19:25:22] code 400, message Bad request syntax ('AT+CIPSEND=33')
xxx.xxx.xxx - - [21/Sep/2018 19:25:22] "AT+CIPSEND=33" 400 -
xxx.xxx.xxx - - [21/Sep/2018 19:25:35] code 400, message Bad request syntax ('AT+CIPSEND=33')
xxx.xxx.xxx - - [21/Sep/2018 19:25:35] "AT+CIPSEND=33" 400 -
xxx.xxx.xxx - - [21/Sep/2018 19:25:49] code 400, message Bad request syntax ('AT+CIPSEND=33')
xxx.xxx.xxx - - [21/Sep/2018 19:25:49] "AT+CIPSEND=33" 400 -

My code accepts the value in the parameter format: Code :

from flask import Flask, jsonify, request
app = Flask(name)

@app.route('/json/',methods=['GET'])
def search(data):
with open("json.txt", "a") as myfile:
myfile.write(str(data)+ "\n");
return jsonify({'message' : 'It works'})

if name == 'main':
app.run(host='0.0.0.0',debug=True,port=80)

Sending the same command in the code as serial monitor but doesnt work

you wont get happy with softserial at 115200 and ESP in AT mode on long term.

just as reminder after two days of not working :

noiasca:
b) change your implementation and programm the ESP.

Thanks for helping me through out. I had got it worked with minor changes. I also agree with you that it is not the best way to send the data. Can you please help me further on best way or recommended way to send data over the internet.

I am now looking at mqtt protocol libraries to connect to aws IOT for which I can use aws sdk. However for connecting to internet, on researching bit more I found out there are libraries available to connect to internet. For instance inbuilt wifi and ESP8266WiFi and some other custom libraries. I understand that these libraries are built to run with WIFI shield / ethernet shield and not esp8266 module seprately, how I have interfaced with arduino (Please correct me if I am wrong).

If you are aware of any library that can be used to directly interface ESP8266 module with arduino to connect to internet without making me purchase ESP8266 wifi shield or ethernet shield

Thanks again, it is really cool to find people are helping here others.

-Pratik