ESP8266 tcp to serial passthrough - how to turn off characters being echoed?

Projects Programming
1

I am working on a project to provide WiFi capability for an Arduino using an ESP8266 module, which connects to the Ardiuino via serial. The Arduino has a simple command interface that allows the user to type a command in a terminal and the Arduino responds accordingly. It works fine over USB and Bluetooth.

As part of this WiFi connectivity, I have set up rudimentary TCP passthrough to the serial port. This does work, but I am having some difficulty with reliable transmission. I am using PuTTY for the terminal and there is an inconsistency in that characters get echoed when using a TCP connection although I have never had this happen over serial. I can deal with this by forcing echo off in PuTTY, but wondered whether I need to force echo off in the WiFi client somewhere?

Secondly, when typing in a TCP terminal, the first typed command sequence is always missed. It works fine from the second one onwards. This suggests that maybe one or more initial characters are being lost or the action of starting to type 'wakes up' the connection somehow and it only starts accepting input after the first CRLF.

My code is below. Can anyone see anything obviously wrong, or suggest why I get no response to the fist typed sequence?

This version uses buffering:

#include <EEPROM.h>
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>

/***** Web server and client objects *****/
ESP8266WebServer *AR488srv = new ESP8266WebServer(AP.webp);
WiFiServer *passSrv = new WiFiServer(AP.gpibp);
WiFiClient passCli;

/***** Buffers *****/
// HTML page buffer
const uint16_t htmlSize = 4096;
char html[htmlSize];
// Serial buffer
const uint16_t sbSize = 64;
uint8_t sBuf[sbSize];
uint16_t sbPtr = 0;
/***** Buffers *****/


void setup() {

  Serial.begin(BAUD);

#ifdef DEBUG_0
  Serial.println();
  Serial.println(F("Starting WiFi..."));
#endif

  // Set defaults
  setWifiDefault();

  // Start WiFi
  startWifi();

  // Start webserver
#ifdef DEBUG_0  
  Serial.println(F("Starting webserver..."));
#endif  
  startWebServer();

  passSrv->begin();

}


void loop() {

  // Handle requests for web server
  AR488srv->handleClient();

  // Is GPIB passthrough enabled?
  if (AP.gpib) {
    // Handle incoming client connections
    if (passCli.connected()) {
      // Handle connection
      memset(sBuf,'\0',64);
      while(passCli.available()){
        if (sbPtr<sbSize-1){
          sBuf[sbPtr] = passCli.read();
          sbPtr++;
        }
      }
      Serial.write(sBuf, sbPtr);
      sbPtr = 0;

      memset(sBuf,'\0',64);
      while(Serial.available()) {
        if (sbPtr<sbSize-1){
          sBuf[sbPtr] = Serial.read();
          sbPtr++;
        }
        passCli.write((char*)sBuf, sbPtr);
        sbPtr = 0;
      }
 
    }else{
      // Wait for a connection
      passCli = passSrv->available();
    }
  }

}

I have also tried this simpler unbuffered approach but with the same result:

#include <EEPROM.h>
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>

***** Web server and client objects *****/
ESP8266WebServer *AR488srv = new ESP8266WebServer(AP.webp);
WiFiServer *passSrv = new WiFiServer(AP.gpibp);
WiFiClient passCli;

/***** Buffers *****/
// HTML page buffer
const uint16_t htmlSize = 4096;
char html[htmlSize];
// Serial buffer
const uint16_t sbSize = 64;
uint8_t sBuf[sbSize];
uint16_t sbPtr = 0;
/***** Buffers *****/


void setup() {

  Serial.begin(BAUD);

#ifdef DEBUG_0
  Serial.println();
  Serial.println(F("Starting WiFi..."));
#endif

  // Set defaults
  setWifiDefault();

  // Start WiFi
  startWifi();

  // Start webserver
#ifdef DEBUG_0  
  Serial.println(F("Starting webserver..."));
#endif  
  startWebServer();

  passSrv->begin();

}


void loop() {
  // Handle requests for web server
  AR488srv->handleClient();
  // Is GPIB passthrough enabled?
  if (AP.gpib) {
    // Handle incoming client connections
    if (passCli.connected()) {
      // Handle connection
      while(passCli.available()){
        Serial.write(passCli.read());
      }
      Serial.write(sBuf, sbPtr);
      sbPtr = 0;

      memset(sBuf,'\0',64);
      while(Serial.available()) {
        passCli.write(Serial.read());
      }
 
    }else{
      // Wait for a connection
      passCli = passSrv->available();
    }
  }
}

I have also searched the documentation, but can't find any reference to turning the echoing of characters off or character loss.

2

I seem to have isolated the response problem although I do not know what is causing it. Observing the output from the ESP8266 serial via an FTDI UART board, it became evident that as the TCP connection is established over WiFi, some spurious characters are randomly printed to the serial port. The first line of my input typed in at the terminal gets preceded with these random characters and so the parser running on the Arduino board can't make sense of the input and so I don't get the expected response.

Is there any way of preventing these spurious characters from being generated when the TCP connection is established?

3

Ok, I have solved the spurious characters problem by replacing this:

   }else{
      // Wait for a connection
      passCli = passSrv->available();

with this:

} else {
// Wait for a connection
passCli = passSrv->available();
if (passCli.connected()) {
// Initialise buffer
clrSBuf(64);
delay(50);
// Clear spurious characters after connection established
while (passCli.available()) { passCli.read(); }
}
}

The short 50msec delay allows enough time for the spurious characters to accumulate as the tcp connection is being established. The while loop immediately after that clears those waiting spurious characters before we accept any input. The process then continues around the loop to the 'if passCli.connected() section whether the passed through data is processed.

I initially tried passCli.flush() but quickly realised that this works in the same way as Serial.flush() which clears outgoing rather than incoming data in the buffer.

4

How are you connecting to it with PuTTY? Are you using the "Telnet" connection type? That would likely cause PuTTY to send "spurious" bytes at the start of a connection. If that's the case, change the connection type to "Raw".

5

Yes, I was using a standard Telnet connection. Thank you for your suggestion, however, I would be interested to know why that would cause PuTTY to send spurious bytes? I wasn't aware that Telnet sent any pre-amble information. I will try RAW and report back although I will leave my solution in place though as that is the first option users are likely to try.

EDIT: I commented out my solution and tested a RAW connection. This does indeed connect without producing any spurious characters so your observation was spot on. Thanks.

6

BitSeeker:
Yes, I was using a standard Telnet connection. Thank you for your suggestion, however, I would be interested to know why that would cause PuTTY to send spurious bytes?

Telnet has various options, such as to set the character set, echo on/off, etc. PuTTY sends a small number of options when it connects to a Telnet server.

The version of PuTTY I have on my computer (0.70) sends these 21 bytes: ff fb 1f ff fb 20 ff fb 18 ff fb 27 ff fd 01 ff fb 03 ff fd 03. Those are decoded as:

  • ff fb 1f WILL Negotiate About Window Size
  • ff fb 20 WILL Terminal Speed
  • ff fb 18 WILL Terminal Type
  • ff fb 27 WILL New Environment Option
  • ff fd 01 DO Echo
  • ff fb 03 WILL Suppress Go Ahead
  • ff fd 03 DO Suppress Go Ahead
7

Thank you for the detailed replay which was educational. I was getting around 18-19 bytes, so not far off your 21. Could be I miscounted as well, but interesting to note that Telnet communicates its intention to the host upon establishing a connection in this way.

I will, of course make a note to use a RAW connection for my app. Really appreciate your help. :slight_smile: