WifiESP library and data errors at 115200 bps

This is a question for those contributors to this forum who have commercial experience with this sort of stuff I suspect.

I have linked the ESP8266 Tx and Rx pins to Rx1 and Tx1 on my Mega with to short lengths of wire with one of those female header socket things at each end.

I then twisted the wires a bit to hold them together.

Then I got some copper take and wrapped it around most of the length and then connected this to GND via another short length of wire - some crude RF shielding.

But I am still getting data errors, e.g. a lot of missing characters, from my http headers etc.

BUT I am uploading sketches to my Mega with a MUCH longer USB cable and at 115200 bps with no problems at all.

So my question is what would it take to get this reliability between my ESP8266 and my Mega through much shorter cables?

What type of cable would the experts suggest I try? Something small and relatively inexpensive.

Thought of cannibalizing one of my surplus arduino USB cables but even they are going to be fairly chunky for the length I need, and they have two extra Vcc and GND conductors I don't need.

Any suggestions?

boylesg:
This is a question for those contributors to this forum who have commercial experience with this sort of stuff I suspect.

I have linked the ESP8266 Tx and Rx pins to Rx1 and Tx1 on my Mega with to short lengths of wire with one of those female header socket things at each end.

I then twisted the wires a bit to hold them together.

Then I got some copper take and wrapped it around most of the length and then connected this to GND via another short length of wire - some crude RF shielding.

But I am still getting data errors, e.g. a lot of missing characters, from my http headers etc.

BUT I am uploading sketches to my Mega with a MUCH longer USB cable and at 115200 bps with no problems at all.

So my question is what would it take to get this reliability between my ESP8266 and my Mega through much shorter cables?

What type of cable would the experts suggest I try? Something small and relatively inexpensive.

Thought of cannibalizing one of my surplus arduino USB cables but even they are going to be fairly chunky for the length I need, and they have two extra Vcc and GND conductors I don't need.

Any suggestions?

You are missing received characters?

It sounds like it is a software issue. The Standard Arduino environment allocates 2 x 64 byte for buffers for each Serial object, one for the TX one for the RX. If you are missing characters, I suspect you are not read the Serial1() fast enough.

I simple way to see if you are dropping characters:

#define HIGHWATER 60  //adjust this up and down to see where the buffered level maxes out at 
#define LEDPIN 13

pinMode(LEDPIN,OUTPUT);

uint16_t overflow=0;


// include this if before your Serial1.read(), it is quick  
If(Serial1.available()>HIGHWATER) {
  overflow++; 
  digitalWrite(LEDPIN,HIGH);
  }
else digitalWrite(BUILTINLED,LOW);

// if the led is blinking you are not servicing the Serial1() object often enough.

at 115,200 baud, 11,520cps, 60 characters are received in 5.2ms, enough time for your 16MHz Mega to execute 83k instruction. You must service the Serial1() object at least once every 5ms.

Chuck.

chucktodd:
You are missing received characters?

It sounds like it is a software issue. The Standard Arduino environment allocates 2 x 64 byte for buffers for each Serial object, one for the TX one for the RX. If you are missing characters, I suspect you are not read the Serial1() fast enough.

I simple way to see if you are dropping characters:

#define HIGHWATER 60  //adjust this up and down to see where the buffered level maxes out at 

#define LEDPIN 13

pinMode(LEDPIN,OUTPUT);

uint16_t overflow=0;

// include this if before your Serial1.read(), it is quick 
If(Serial1.available()>HIGHWATER) {
  overflow++;
  digitalWrite(LEDPIN,HIGH);
  }
else digitalWrite(BUILTINLED,LOW);

// if the led is blinking you are not servicing the Serial1() object often enough.




at 115,200 baud, 11,520cps, 60 characters are received in 5.2ms, enough time for your 16MHz Mega to execute 83k instruction. You must service the Serial1() object at least once every 5ms.

Chuck.

OK, makes sense.

But to interpret the form data correctly I am obligated to read the port on char at a time - I need to pick up '%' in order to convert it and the following hex value to the corresponding ASCII character.
This is because I am sending lists of dates and times as values for some hidden form fields.

I ended up reducing the baud rate of the ESP8266 to 74880 bps and I seem to have solved this problem.

However I now have another:

The following are HTTP requests received and dumped in the Serial monitor.

The default HTTP requests 'GET / HTTP/1.1' seem to work just fine.

But as soon as I try and send through a POST request 'POST /program.htm HTTP/1.1' I am getting a timeout error after my post data appears to have been successfully read:

"R=&euny&L=TS=&NRT=&TSD=&Station1=suspend%3A%231%2F1+-+00%3A00%2810%29%231%2F5+-+00%3A00%2810%29%231%2F9+-+00%3A00%2810%29&Station2=suspend%3A%23&Station3=suspend%3A%23&Station4=suspend%3A%23&Station5=suspend%3A%23&Station6=suspend%3A%23&Station7=suspend%3A%23&Station8=suspend%3A%23[WiFiEsp] TIMEOUT: 73"

Changing the timeout values in EspDrv.cpp from the WifiESP library makes no difference.

I notice that there is no \r\n\r\n after my data and I am wondering if bool EspDrv::getData(uint8_t connId, uint8_t data, bool peek, bool connClose) in EspDrv.cpp is expecting this in order to terminate the loop successfully?

And I am also wondering if EspDrv::getData(uint8_t connId, uint8_t data, bool peek, bool connClose) is designed specifically for GET requests only and not POST request?

[WiFiEsp] New client 0
New HTTP request
GET / HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

*************************
HTTP request:
GET / HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1


*************************
Sending default HTTP response...
RTC battery voltage: 3.4, 351
HTTP response took 17886ms to service...
[WiFiEsp] Disconnecting  0
Client disconnected
[WiFiEsp] New client 0
New HTTP request
POST /program.htm HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://10.0.0.79/
Connection: kee-lv
UgaeneueRqet:1CnetTp:plcto/wwfr-rnoe
Ctn-et:25
R=&euny&L=TS=&NRT=&TSD=&Station1=suspend%3A%231%2F1+-+00%3A00%2810%29%231%2F5+-+00%3A00%2810%29%231%2F9+-+00%3A00%2810%29&Station2=suspend%3A%23&Station3=suspend%3A%23&Station4=suspend%3A%23&Station5=suspend%3A%23&Station6=suspend%3A%23&Station7=suspend%3A%23&Station8=suspend%3A%23[WiFiEsp] TIMEOUT: 73

boylesg:
OK, makes sense.

But to interpret the form data correctly I am obligated to read the port on char at a time - I need to pick up '%' in order to convert it and the following hex value to the corresponding ASCII character.
This is because I am sending lists of dates and times as values for some hidden form fields.

I ended up reducing the baud rate of the ESP8266 to 74880 bps and I seem to have solved this problem.

However I now have another:

The following are HTTP requests received and dumped in the Serial monitor.

The default HTTP requests 'GET / HTTP/1.1' seem to work just fine.

But as soon as I try and send through a POST request 'POST /program.htm HTTP/1.1' I am getting a timeout error after my post data appears to have been successfully read:

"R=&euny&L=TS=&NRT=&TSD=&Station1=suspend%3A%231%2F1+-+00%3A00%2810%29%231%2F5+-+00%3A00%2810%29%231%2F9+-+00%3A00%2810%29&Station2=suspend%3A%23&Station3=suspend%3A%23&Station4=suspend%3A%23&Station5=suspend%3A%23&Station6=suspend%3A%23&Station7=suspend%3A%23&Station8=suspend%3A%23[WiFiEsp] TIMEOUT: 73"

Changing the timeout values in EspDrv.cpp from the WifiESP library makes no difference.

I notice that there is no \r\n\r\n after my data and I am wondering if bool EspDrv::getData(uint8_t connId, uint8_t data, bool peek, bool connClose) in EspDrv.cpp is expecting this in order to terminate the loop successfully?

And I am also wondering if EspDrv::getData(uint8_t connId, uint8_t data, bool peek, bool connClose) is designed specifically for GET requests only and not POST request?

[WiFiEsp] New client 0

New HTTP request
GET / HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1


HTTP request:
GET / HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1


Sending default HTTP response...
RTC battery voltage: 3.4, 351
HTTP response took 17886ms to service...
[WiFiEsp] Disconnecting  0
Client disconnected
[WiFiEsp] New client 0
New HTTP request
POST /program.htm HTTP/1.1
Host: 10.0.0.79
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://10.0.0.79/
Connection: kee-lv
UgaeneueRqet:1CnetTp:plcto/wwfr-rnoe
Ctn-et:25
R=&euny&L=TS=&NRT=&TSD=&Station1=suspend%3A%231%2F1+-+00%3A00%2810%29%231%2F5+-+00%3A00%2810%29%231%2F9+-+00%3A00%2810%29&Station2=suspend%3A%23&Station3=suspend%3A%23&Station4=suspend%3A%23&Station5=suspend%3A%23&Station6=suspend%3A%23&Station7=suspend%3A%23&Station8=suspend%3A%23[WiFiEsp] TIMEOUT: 73

GET and POST transmissions are very similar, GET does not have a body, and parameters are passed by appending them to the web page: See Wiki

GET \pagename?fieldname1=data&fieldname2=data HTTP/1.1
Host: hostname

terminated with a blank line

While a POST moves the parameters into a body:

POST \pagename HTTP/1.1
Host: hostname 

fieldname1=data&fieldname2=data

a blank line separates the BODY from the HEADER, and the BODY is Terminated by a blank line. The BODY could be MIME encoded.

It still looks like you are loosing data.

The Post Data should be:

fieldname1=data&fieldname2=data ...

And, many symbols are % escaped. So If i take your example output

R=&euny&L=TS=&NRT=&TSD=&Station1=suspend%3A%231%2F1+-+00%3A00%2810%29%231%2F5+-+00%3A00%2810%29%231%2F9+-+00%3A00%2810%29&Station2=suspend%3A%23&Station3=suspend%3A%23&Station4=suspend%3A%23&Station5=suspend%3A%23&Station6=suspend%3A%23&Station7=suspend%3A%23&Station8=suspend%3A%23

Becomes:

R=
euny           // this one is bad
L=TS=        // this one is bad
NRT=
TSD=
Station1=suspend:#1/1+-+00:00(10)#1/5+-+00:00(10)#1/9+-+00:00(10)
Station2=suspend:#
Station3=suspend:#
Station4=suspend:#
Station5=suspend:#
Station6=suspend:#
Station7=suspend:#
Station8=suspend:#

How much of this decoded data is what you exected?

The [WiFiEsp] TIMEOUT: 73 message just means that you did not acknowledge the post data to the client within the timeout period.

Code to accept these parameters into a memory buffer with null terminators would be something like this:

#define BUFFLEN 500      // pool for parameters

char pool[BUFFLEN+1]; // + room for final NULL
uint16_t poolPos=0;    // next position in pool

bool done=false;
bool blankline=false;
bool error=false;

while(!done&&!error){
  if(Serial.available()){
    ch=Serial.read();
    if(ch=='%'){ // escaped char
      while(!Serial.available()); // wait forever for char,  needs a timeout!
      ch=Serial.read()-48;
      pool[poolPos] = (ch>9?ch-7:ch)*16; // convert from hex to Decimal
      while(!Serial.available()); // wait forever for char,  needs a timeout!
      ch=Serial.read()-48;
      pool[poolPos++] += (ch>9?ch-7:ch); // convert from hex to Decimal
      }
    else if (ch=='\n' && blankline){ // done, Yea
      done = true;
      pool[poolPos++]= '\0'; // terminal null for last param
      }
    else if (ch=='\n') { // starting new line
      blankline = true;
      }
    else if (ch!='\r') { // non blank line
      blankline=false;
      if(ch=='&'){ // separator between params
        pool[poolPos++]='\0';
        }
      else pool[poolPos++]=ch;
      }
    error=!(poolPos<BUFFLEN); // need room for two NULLs, one for the last string, one for the buffer
    }
  // should have some timeout
  }
if(!error) pool[poolPos]='\0';  // Make a double NULL to mark end of params in buffer
// Also, pooPos will indicate end of the 'used' pool.

if (done) send_http_acknowledge (200);

pool[] will have all of the parameters a cstrings, each cstring will be name=value

Chuck.