ESP8266 as a serial port server, how to save parameters?

This is my first post on this forum, I just joined, so please excuse me if the questions are already answered! I have tried googling a lot and spent a few complete days worth of reading for this project, but now I need somewhere to ask too....

I need to design an application for the ESP8266 (the ESP-07S module) in order to deliver the following functions:

  • ESP-07S acting as a WiFi AP with protected login
  • Serial port server (completely transparent) on TCP port 2001, binary data in/out
  • Simple configuration web page on TCP port 80 (or maybe another more obscure port...)

The configuration page shall contain the following configurable items:

  • SSID for the AP
  • Password for the AP
  • Baudrate for the serial port
  • Changes saved using a password protected button

This solution is in order to expose the serial connection for a data collection instrument onto a WiFi network interface. PC:s and Android devices shall be able to talk to the instrument via TCP rather than RS232 with this adapter.
We already did this back in 2013 using a commercial WiFi module pre-programmed with the serial server etc. But now the module is discontinued and I felt that rather than get into trouble with another supplier we should do it ourselves using readily available IoT WiFi modules.

QUESTIONS:

1) Configuration store
Main issue for me is how to save and read back the modified data entered on the config webpage.
If this is not possible then the ESP-07S cannot be used because the system is frequently power cycled and the AP should stay at the same SSID/Passwrd all the time.
Same for the baud rate (19200 and 38400 are the alternatives).

2) Example code for serial port server?
If someone has a good sketch to start from I would be happy to get a link to the source-code to which I can add the web interface. I have found one for the NodeMCU interface but I'd much rather use the Arduino IDE and compile a complete image. Should give much better performance than interpreted solutions.

3) Simple webpage example
Is there some good example of a basic webpage with entry boxes and a button for activation requiring some kind of activation code?

I have seen examples of actions like toggling LED:s etc, but none thta saves data into some permanent location and applies the data to the running process.
Also I have not seen any webserver sketch that can handle both a config page and an active serial port server...
My main issue is making settings changes persistent.

with esp8266 arduino core you can write an arduino sketch for the esp8266.

install it and install WiFiManager library and look at examples.

I have a more complex example of esp8266 skeleton sketch with my TelnetStream library

Thanks,
I looked at the TelnetStream but was none the wiser...

One thing I really should have said more clearly is that the serial port server must handle all bytes without exception.
In the discussions I have seen on the web it seems like methods stop at the first NULL character (0x00) in the data stream. This must not happen because the transfers are totally binary and there are lots of 0x00 inside. All bytes have an equal chance of being part of a transfer...

And I should also have pointed out that the ESP-07S will be the ONLY device in the system, it is not a peripheral to an Arduino board. So the CPU of the ESP-07S is what has to run it all...

I have seen the description of the ESP8266 library but it does not say how much space there is, unfortunately.
How big of a space is there?
I assume that I can use the EEPROM functions to save configuration to flash?
Finally: Is it enough to mention #include <ESP8266WiFi.h> at the top of the sketch file in order for the EEPROM functions to be available? (Remember I'm a newbie to Arduino/ESP8266)

BosseB:
Thanks,
I looked at the TelnetStream but was none the wiser...

It is a skeleton sketch with common services. Nothing more.

BosseB:
One thing I really should have said more clearly is that the serial port server must handle all bytes without exception.
In the discussions I have seen on the web it seems like methods stop at the first NULL character (0x00) in the data stream. This must not happen because the transfers are totally binary and there are lots of 0x00 inside. All bytes have an equal chance of being part of a transfer...

It will work how you program it. Do not trust what you read on Internet.

BosseB:
And I should also have pointed out that the ESP-07S will be the ONLY device in the system, it is not a peripheral to an Arduino board. So the CPU of the ESP-07S is what has to run it all...

I did not assume other. esp8266 has enough power to copy serial data from WiFi to TTL

BosseB:
I have seen the description of the ESP8266 library but it does not say how much space there is, unfortunately.
How big of a space is there?

space where? esp8266 uses external flash memory for program code and file system. 4MB is common flash size, but check before buying. 4Mb is common too.

BosseB:
I assume that I can use the EEPROM functions to save configuration to flash?

other possibility is save to file in the file system

BosseB:
Finally: Is it enough to mention #include <ESP8266WiFi.h> at the top of the sketch file in order for the EEPROM functions to be available? (Remember I'm a newbie to Arduino/ESP8266)

there is a EEPROM library which emulates EEPROM in external flash memory

they are so cheap. buy a wemos d1 mini and try it

Juraj:
space where? esp8266 uses external flash memory for program code and file system. 4MB is common flash size, but check before buying. 4Mb is common too.

Well I meant the size in bytes of the "EEPROM"

other possibility is save to file in the file system

I did not know there is a file system on the ESP8266!
If there is that would be a "normal" way of storing configs, I guess.
I have to try and find some more info regarding this....

they are so cheap. buy a wemos d1 mini and try it

Well, I need a small module I can put on our board to replace the discontinued WiFi module.
So a prototyping board like Wemos D1 Mini is not that.
The AI-Thinker ESP-07S is apparently filling our needs:

  • Small outline
  • Shielded design
  • Coaxial connector for antenna
  • No built-on antenna
  • Easy to connect into a redesigned board.
    Our current board looks like below (size 55x35 mm), the marked sub-board is going to be replaced by an ESP-07S soldered directly to our board after it has been slightly redesigned.

the Wemos only as a development board. to get know the posibilities.

esp8266 doesn't have build-in EEPROM.

Sounds like esp-link does most of what you want.

I have looked at EEPROM now and I have a question regarding its use:
As I understand it EEPROM.h defines methods to handle the EEPROM on an Arduino board.
When the ESP8266WiFi library is used then it seems like EEPROM functions a a bit differently depending on the ESP8266 internal layout which does not include a dedicated EEPROM, instead some part of flash is used as a simulated EEPROM.
This brings up a C language issue, the order of declarations of include files...
Which is correct:

#include <ESP8266WiFi.h>
#include <FS.h>
#include <EEPROM.h>

or

#include <EEPROM.h>
#include <ESP8266WiFi.h>
#include <FS.h>

I.e. does the final result depend on how the include statements are ordered?
Same for FS.h...

I do not understand. There is no such problem. Those are esp8266 versions of the libraries from the esp8266 package.

I linked esp8266 core installation instructions in replay #1. Install it into IDE, select some esp8266 board and you can start write and compile sketches.

Sorry, I am used to the case where #include file order being important if they define the same function or other object.
Then the last included one "wins". Since the normal Arduino EEPROM manual page does not list the EEPROM.commit() or EEPROM.end() methods I worried that listing the include files in the wrong order might cause problems, assuming EEPROM is redeclared inside the ESP8266WiFi.h file.
But maybe this is solved behind the scenes by telling Arduino about the board used and then things are resolved without the developer having to worry....

In any case I have now started to modify a sketch I used a year ago when I first briefly touched the ESP8266 and I have a question regarding the port server included in this (I have not yet come as far as being able to start it up on real hardware):

As I have stated I want the configuration to be accessible via a webpage and on this it should be possible to change the following items on the system:

  • AP SSID
  • AP Password
  • AP IP address
  • Serial port server TCP port
  • Serial port server baudrate
  • Config webpage TCP port number (should be possible to hide it behind a non-obvious port)

I think I have mastered the way to read the values from EEPROM now (using EEPROM.get() and EEPROM.read() methods. If they are not present/valid I am letting the setup() function write the defaults into EEPROM too.

But concerning the serial port server it seems like it is not possible to read the port number from EEPROM before it is initialized. The examples I have found show that the server object is not created inside a function but in the global variable declarations, something like this:

#include <ESP8266WiFi.h>
#include <FS.h>
#include <EEPROM.h>

#define TCP_PORT  2001    // Choose any port you want

WiFiServer tcpServer(TCP_PORT); //<= I need TCP_PORT to be configured before use
.....
void setup() {
//Read config from EEPROM and start up system
....
}
void loop() {
//Main function here
}

I want to change the TCP-PORT from being a define to being an unsigned int global variable, which can be initialized in the setup() function, but that seems not possible because it is used already before reaching setup()...
Can the WiFiServer object be instantiated without setting the port number and instead do this later inside setup()?
If so how could that be done?
I could not find any properties or methods to use to set the port number for the server....

I finally found a web discussion from 2016 where others have discussed the same problem of setting the TCP port of an instantiated WiFiServer and where there is a solution too!
Apparently there has been a change in the way it operates such that this sequence is allowed:

unsigned int TCP_PORT = 2001;
WiFiServer tcpServer(TCP_PORT);

void setup() {
  LoadConfig();  //Read new or old value of TCP_PORT
  tcpServer.begin(TCP_SERVER);   //closes if running, then opens again on the new port.
  ....

When looking at the documentation for server.begin() there is no mention of any port parameter....
I still cannot test this but now I have come a bit further, unless of course I have to do something special to make the begin parameter work...

there is a library for the configuration of persistent WiFi parameters. WiFiManager

Now I have come as far as I can "Verify" the sketch in Arduino and after I have cleaned up some other errors I have reached the thing this thread is all about.
I get an error saying:

TCP-UART_Bridge:185: error: no matching function for call to 'WiFiServer::begin(unsigned int&)'

It appears on the line where I want to start the server with the correct port number (see snippet below):

#include <ESP8266WiFi.h>
#include <FS.h>
#include <EEPROM.h>

IPAddress apIP(192, 168, 211, 1);
String APSSID = "SS9999";  // Set default SSID
String PASSWD = "SS123_xx01"; // minimum 8 characters
unsigned int TCP_PORT = 2001;    // Choose any port you want
WiFiServer tcpServer(TCP_PORT);
... code ....
void setup(){
  ...other code...
  WiFi.mode(WIFI_AP);
  WiFi.softAPConfig(apIP, apIP, IPAddress(255, 255, 255, 0));
  WiFi.softAP(APSSID.c_str(), PASSWD.c_str());
  // Restart TCP listener on port TCP_PORT
  tcpServer.begin(TCP_PORT); // <== Error here!
  tcpServer.setNoDelay(true);

According to the web discussion from 2016 I linked to above the begin() method is expanded to include an optional parameter for the port, but that seems not to be the case for me. Why?
Or do I have to do something special in order to make it work? Updating some library, perhaps? In that case, how is that done?
I am using Arduino 1.8.2 if that matters.

There isn't a begin() with port as parameter. You can create the server object in setup().

WiFiServer *telnetServer;

void setup () {
  telnetServer = new WiFiServer(23);
  telnetServer->begin();

Thanks for your suggestion!
I had not thought of declaring the tcpServer as a pointer...

So I have modified the code as shown below and ran a Verify in Arduino.
Result: it completes without errors! :slight_smile:

...
WiFiServer *tcpServer;
...
void setup() {
  EEPROM.begin(512); //Initialize EEPROM to load from flash and write changes later on
  LoadConfig();  //Populates init data from EEPROM

  WiFi.mode(WIFI_AP);
  WiFi.softAPConfig(apIP, apIP, IPAddress(255, 255, 255, 0));
  WiFi.softAP(APSSID.c_str(), PASSWD.c_str());

  //start UART. Be sure to set the speed to match the speed of whatever is
  //connected  to the UART.
  SerialSS.begin(BaudRate);

  // Instantiate and start TCP server on port TCP_PORT
  tcpServer = new WiFiServer(TCP_PORT);
  tcpServer->begin();
  tcpServer->setNoDelay(true);
  ...
}
void loop() {
  uint8_t i;
  char buf[1024];
  int bytesAvail, bytesIn;

  //check if there are any new clients
  if (tcpServer->hasClient()) {
    for (i = 0; i < MAX_SRV_CLIENTS; i++) {
      //find free/disconnected spot
      if (!tcpServerClients[i] || !tcpServerClients[i].connected()) {
        if (tcpServerClients[i]) tcpServerClients[i].stop();
        tcpServerClients[i] = tcpServer->available();
        SerialDebug.print("New client: "); SerialDebug.print(i);
        continue;
      }
    }
    //no free/disconnected spot so reject
    WiFiClient tcpServerClient = tcpServer->available();
    tcpServerClient.stop();
  }

  //check clients for data
  for (i = 0; i < MAX_SRV_CLIENTS; i++) {
    if (tcpServerClients[i] && tcpServerClients[i].connected()) {
      //get data from the serial client and push it to the UART
      while ((bytesAvail = tcpServerClients[i].available()) > 0) {
        bytesIn = tcpServerClients[i].readBytes(buf, min(sizeof(buf), bytesAvail));
        if (bytesIn > 0) {
          SerialSS.write(buf, bytesIn);
          delay(0);
        }
      }
    }
  }

  //check UART for data
  while ((bytesAvail = SerialSS.available()) > 0) {
    bytesIn = SerialSS.readBytes(buf, min(sizeof(buf), bytesAvail));
    if (bytesIn > 0) {
      //push UART data to all connected serial clients
      for (i = 0; i < MAX_SRV_CLIENTS; i++) {
        if (tcpServerClients[i] && tcpServerClients[i].connected()) {
          tcpServerClients[i].write((uint8_t*)buf, bytesIn);
          delay(0);
        }
      }
    }
  }
}

I am not familiar with the -> notation replacing the . notation, but it is accepted by the compiler now.

So next step is to arrange for some support stuff to run a test, which will require a TCP socket application on the PC side working like the Termite console I used with the AT interface to the ESP8266 module over the serial line.
I could maybe use PuTTY for that...
Then I also need to set up the webserver for configuration data entry of course.
But that is another story...

Thanks again for your suggestions, they have helped a lot in combination with Google!

PS: Do you know why I do not receive any alert emails for replies to this thread?
I have set the "RECEIVE EMAILS" box, but I get no alerts anyway..
I am new to this forum too.... DS

look at WiFiManager library

object.member
pointer-to-object -> object-member

Alerts Settings link is in Alerts pop-up. default is no alerts

Now I am back with a stupid question:
I grabbed this EEPROM data reading code from an instruction page:

  #define SSID_ADDR 0 
  String essid = "";
  int i;
  EEPROM.begin(512); //Initialize EEPROM to load from flash and write changes later on
  //Read AP SSID value from EEPROM
  for (i = 0; i < 32; ++i)
      essid += char(EEPROM.read(i + SSID_ADDR));

But now when I look at it it seems a bit odd because of the ++i in the loop...
Should it not be i++ instead?
I.e. does the EEPROM address space start at 0 or 1?

BosseB:
I.e. does the EEPROM address space start at 0 or 1?

0

After some googling I found that in simple for loops like:

for (i=0; i<endvalue; ++i) do_something(i);

the result of using ++i and i++ is the same since the increment statement itself does nothing with the value of i.
But there are apparently performance reasons for using ++i when programming in C++ as opposed to C (I have never ever programmed C++ myself)...
See this pretty long (and animated) discussion at Stackoverflow!
So I will keep ++i since I have understood that beneath the hood Arduino uses C++ even though it is not exposed in the ino file.
And thanks for confirming my belief that EEPROM addresses are zero based!

PS: I now found that the Arduino IDE editor seems to deviate from what all other editors I have used does when using Ctrl-Z! In these the Ctrl-Z key reverts the last edit operation but when I made a typo in Arduino IDE and used Crtl-Z then suddenly ALL of my edits from the last save to disk including the change to using a pointer variable for tcpServer were removed!
Is this normal and is there a configuration to return to a sensible Ctrl-Z operation? DS

Arduino is C++ with some shallow .ino handling

my installation of Arduino IDE 1.8.5 has usual Undo function with Ctrl+Z