More correct way to send and receive data over WiFi using HTTP between ESP8266

Hi,
for communication between ESP8266 devices, I have so far used the simplest method, i.e. communication over the HTTP protocol using a specific link. Unfortunately, this solution (perhaps incorrectly written and modified by me) sometimes causes some problems.

  1. First of all, it happens that another ESP8266 module is turned off for some time, so that the HTTP request is sent to the ESP8266 module not connected at the moment - this causes very long delays and sometimes a restart of the ESP2866. I send a lot of such requests (even 4 every 500ms). Is there a solution to this problem? Perhaps imposing a smaller timeout while waiting to receive data (for example, 300ms)?
  2. Another problem is how I process the received content return data. For the moment, I am using the method I show below (using the GETRequestData[] array). I think this method is not good, it is inefficient, and it constantly generates heap memory clogging with received Strings of different lengths. Wouldn't it perhaps be better to return the content itself without the code using return? And what if the data is not returned correctly (use return false?)?
    The code:
String GETRequestData[2] = {"", ""};
static int GETGateLampDataArray[5];

void sendGETRequest(String URL) {
  GETRequestData[0] = "";
  GETRequestData[1] = "";
  if (WiFi.status() == WL_CONNECTED) {           // WiFi connection check
    WiFiClient client;
    HTTPClient http;
    http.begin(client, URL.c_str());
    int httpResponseCode = http.GET();        // Send GET
    if (httpResponseCode > 0) {
      GETRequestData[0] = String(httpResponseCode);
      GETRequestData[1] = http.getString();
    } else {
      logThis("HTTP Error (httpResponseCode): " + String(httpResponseCode));
    }
    http.end();
  } else {
    logThis("No WiFi");
  }
}

// Update every 500ms
void updateDataOverHTTP() {
  sendGETRequest("http://192.168.0.66/xyzData");
  GETGateLampDataArray[0] = GETRequestData[1].charAt(0) - '0';     // GETGateLampDataArray[0] - dane o stanie lampek
  GETGateLampDataArray[1] = GETRequestData[1].charAt(2) - '0';     // GETGateLampDataArray[1] - dane o stanie pozycji bramy
  GETGateLampDataArray[2] = GETRequestData[1].charAt(4) - '0';     // GETGateLampDataArray[2] - dane o wartości enkodera
  GETGateLampDataArray[3] = GETRequestData[1].charAt(6) - '0';     // GETGateLampDataArray[3] - dane o stanie blokady bramy
}
  1. Another issue is the question of how to receive data from another device, of which there will be many and will contain values in the form of double. On the device that has the data, I would like to use a function similar to the following. I would like to separate the data with the _ character, for example. But how to receive the later data efficiently and split it into single variables on the ESP8266 device, which will send a query and receive these return values?
    The code I used always to send multiple data:
server.on("/xDataZ", HTTP_GET, [](AsyncWebServerRequest *request) {
    request->send(200, "text/plain", String(volts) + "_" + String(wats) + "_" + String(amps) + "_" + String(hz));
  });

Example test from content:

231.4_2970_11.2_49.99

A longer time ago I was asking about simple TCP-communicationin this thread:

I tested it a little bit but turned back to UDP as UDP is even simpler (at least for me)
I still had trouble to adapt the demo-code.

If somebody can post a link to a demo-code that does plain-simple KISS keep it simple stupid
tcp-data exchange WITHOUT HTTP between two ESP8266 / ESP32

just
tcp.print() / tcp.read()
as a ready to use code where only SSID and password must be adapted and all other details are coded in a way that will guarantee to make it run.
Please post it.

best regards Stefan

Something like

@misiekd

if you want to build your communication based on HTTP imho the easiest option is that you send (POST) your key-value pairs as parameters to the other device. I would use POST to transmit data (in the body) instead of GET (appending to the URL).

Can you make a schematic/drawing which data should be transfered from the client to the server and which data do you expect to get returned from the server to the client?

But why would you want to do this for simple data transfers over TCP? It seems to me that, unless you're dealing with web pages, HTTP just adds unnecessary layers of syntax and structure.

I can't answer on behalf of the TO. However I would use HTTP over TCP when there is already a webserver in place on the data receiver side (the server), because most of the code is already existing, just read the received parameters.

I don't see a compelling use case for having a "bare" TCP connection (i.e. one that does not have HTTP or some other protocol on top of it) behave like a Stream class object. Consider that a typical application might be sending sensor data from multiple nodes to a common master (server). Such data lends itself well to being packetized. So, I'd just take advantage of TCP's already packetized nature and use the functions already provided by the AsyncClient class.

hm I have almost no understanding of how http-post http-get work.
I do not (yet) understand what the advantage of writing medium complex code that requires to write code like server.on-event-handlers

etc. etc. etc.

has over plain KISSASS (Keep It Simple Stupid And Stupid Simple

by using tcp.print() / tcp.read()

I have one idea but I'm not sure
if one microcontroller receives sensordata from multiple other microcontrollers it might be possible by using http to have the data sorted quickly.
With a serial connection a simple µC-ID would be used to sort it.

So - out of having no knowledge about http - I really don't get it what the advantage is.

You would have to write more details and explicitly describe the disadvantage of not using http.

If there are advantages with using http for data-exchange between microcontrollers
NOT with browsers !!

Can you recommend a library that hides away all the server.on-stuff
and has an API that is as simple as

myWhatEverObject.print() / myWhatEverObject.read()

Again:
it might well be that the above described case of data-exchange between microcontrollers
NOT with browsers !!
is not the typical case for http.

Would you mind describe the typical cases in detail where using http is an advantage?

best regards Stefan

You parsed my statement incorrectly. Read it again perhaps without the part in parentheses:

Meaning I do see the advantage of using "bare TCP" without HTTP when you're sending data between microcontrollers and a web server is not involved. What I don't see the advantage of is making the the TCP object look like a Stream object. Here's why:

How about something like ESP-NOW?

@gfvalvo thank you very much for explaining it so far.

By stream-class-object you mean things like Serial.print() etc.?

Aha. Can you - with small effort - post a link to a well explained (=first class detailed explained) example that demonstrates data-exchange by

?

best regards Stefan

Yes. 'Serial' is an object whose class inherits from Stream.

As I just started looking into this, I haven't searched the web for tutorials / examples. There was quite a bit of example code posted in your Thread from Last Year.
Using ideas from that thread, I spent an hour putting together a skeletal example (for ESP32) of what I mean. It compiles and shows the techniques I'm talking about but would need to be fleshed out to produce code that does something useful.

The idea (as I said above) is that there are a number of slave nodes that gather sensor data and periodically send it to the master (server) for processing. The Server takes the data from the nodes and processes it somehow.

I didn't include any details about the sensors or how the data is processed since they're not needed to demonstrate the main concept.

In my mind, the advantage of doing it this way is that you don't need to turn numeric sensor data into character strings for transmission just to parse and convert it back to numeric data in the Server. All the data is sent and received in its native binary.

SLAVE NODE:

#include "Arduino.h"
#include <WiFi.h>
#include <AsyncTCP.h>

struct SensorData_t {
  uint8_t sensorId;
  time_t timeStamp;
  float temperature;
  float humidity;
  float pressure;
};

void connectWiFi();
void handleConnected(void *arg, AsyncClient *client);
void getSensorData(SensorData_t &theData);

IPAddress serverAddress(192, 168, 10, 100);
const uint16_t port = 7050;
bool waitingToConnect = false;
const size_t sensorDataSize = sizeof(SensorData_t);
AsyncClient aClient;

void setup() {
  Serial.begin(115200);
  connectWiFi();
  aClient.onConnect(handleConnected, nullptr);
}

void loop() {
  static uint32_t measuremetPeriod = 10000;
  static uint32_t prevMillis = millis() - measuremetPeriod;
  uint32_t currentMillis = millis();

  if (((currentMillis - prevMillis) >= measuremetPeriod) && !waitingToConnect) {
    waitingToConnect = true;
    prevMillis = currentMillis;
    aClient.connect(serverAddress, port);
  }
}

void handleConnected(void *arg, AsyncClient *client) {
  (void) arg;
  (void) client;
  SensorData_t newData;

  getSensorData(newData);
  if (aClient.space() >= sensorDataSize && aClient.canSend()) {
    aClient.add(reinterpret_cast<const char *> (&newData), sensorDataSize);
    aClient.send();
  }
  aClient.close();
  waitingToConnect = false;
}

void getSensorData(SensorData_t &theData) {
  // read the sensors and put the results in theData
}

void connectWiFi() {
  // Typical ESP32 WiFi Connection code goes here
}

MASTER SERVER:

#include "Arduino.h"
#include <WiFi.h>
#include <AsyncTCP.h>

struct SensorData_t {
  uint8_t sensorId;
  time_t timeStamp;
  float temperature;
  float humidity;
  float pressure;
};

void connectWiFi();
void handleNewClient(void *arg, AsyncClient *client);
void handleData(void *arg, AsyncClient *client, void *data, size_t len);
void handleError(void *arg, AsyncClient *client, int8_t error);
void handleDisconnect(void* arg, AsyncClient * client);
static void handleTimeOut(void* arg, AsyncClient * client, uint32_t time);

const size_t sensorDataSize = sizeof(SensorData_t);
const uint16_t port = 7050;
AsyncServer server(port);

void setup() {
  Serial.begin(115200);
  connectWiFi();
  server.onClient(handleNewClient, nullptr);
  server.begin();
}

void loop() {
}

void connectWiFi() {
  // Typical ESP32 WiFi Connection code goes here
}

void handleNewClient(void *arg, AsyncClient *client) {
  (void) arg;
  (void) client;
  log_i("Client Connected");
  client->onData(&handleData, NULL);
  client->onError(&handleError, NULL);
  client->onDisconnect(&handleDisconnect, NULL);
  client->onTimeout(&handleTimeOut, NULL);

  // Put any special code that needs to run when client connects here
}

void handleData(void *arg, AsyncClient *client, void *data, size_t len) {
  (void) arg;
  (void) client;
  SensorData_t dataPacket;
  if (len == sensorDataSize) {
    log_i("Received Data:");
    memcpy(&dataPacket, data, len);

    // Put code here to process the receieved senosr data
  }
}

void handleError(void *arg, AsyncClient *client, int8_t error) {
  (void) arg;
  (void) client;
  log_i("Client Error: %d", error);

  // Put any special code to handle client errors here
}

void handleDisconnect(void* arg, AsyncClient * client) {
  (void) arg;
  (void) client;
  log_i("Client Disconnected");

  // Put any special code to handle client disconnect here
}

static void handleTimeOut(void* arg, AsyncClient * client, uint32_t time) {
  (void) arg;
  (void) client;
  (void) time;
  log_i("Client ACK Timeout");

  // Put any special code to handle client timeout here
}

Sounds like nards...
mashed them up few days ago..
meaning, nards can talk to each other through the server..
just using cleint write and read, defined packet structure, multi-threaded server with strong db backend..
firmware stored in blob fields, OTA'd through nard server, can store sketches too, with nice looking code view..
got a few reports for the logs and all data can be exported..
panels are a bit stupid now, but that's changing..
current panels will be terminals and new panels are coming..
going to use android tablets for panels, to be configured through server..

i'm building a monster, 6 more months..
if i survive..

~q

that's really nice..

~q

Yes, I need this:

Ok so, I would like to use HTTP communication method, I really like it and it is easy to use but it is in my opinion slow. This is the one problem - delay when sending request from client to microcontrollers. In my home I have one ESP8266 with installed Blynk IoT library which sending request to all other microcontrollers. And these requests are sent sometimes 5 - 10 and more in the future in every second. And this causes that there delay exist. I don't remember how big is the delay but it can be something like 0.1 - 0.2ms.
And... I would like to get data every 1 second but it is not possible because it will be blocked by other sent requests. The ESP8266 Blynk microcontroller is only the client who ask other microcontrollers for data. I like HTTP method but something faster could be better.

I thinkt that this wont be good solution with in my case.

ESP8266 NodeMCU: ESP-NOW Web Server Sensor Dashboard | Random Nerd Tutorials

ESP32: ESP-NOW and Wi-Fi Web Server Dashboard (Arduino) | Random Nerd Tutorials

The Serial Input Basics Tutorial describes how to parse data like this. However, IMO, this is the wrong technique for transferring multiple data elements (especially numeric data) over an inherently packetized channel. See my Post #10 and Post #13 above.

Yes, I agree with this but I don't know better way to send numeric data.
The best idea could be something like that but from client side (client get data from server):