Seeking architecture suggestion for IoT project

I am seeking some opinions on system architecture for an IoT system.

For my project, I have some ESP8266 devices as clients and a Raspberry Pi as the server. I currently have everything programmed so that all communications start with the client devices and communication is only initiated when something of which the client is aware changes.

This works well - in the sense that it keeps communications to an absolute minimum. However, this approach has a major problem, it can be woefully unresponsive. If something of which the server is aware changes, it cannot communicate this to the client until the next time the client initiates communication - and that could be a long time.

There are, AFAIK, two ways to address this issue:

  • Have the client periodically communicate with the server, i.e., polling
  • Have the server initiate communication with the client as needed

I fully understand how to implement the former, but being an old-time programmer, I hate polling - the inefficiency just bothers me. To use the second approach, the only method that come to mind is to do some basic socket-level programming.

My questions for the group:

  • Is there some other approach that I neglected?
  • Is there a better approach than using socket-level programming to have the server initiate communication with the client?
  • Is there an approach whereby the client can keep the channel with the server open? If so, doesn't this approach necessarily limit the scalability of the system?

I look forward to hearing your thoughts on this matter?

gproston:

  • Have the client periodically communicate with the server, i.e., polling
  • Have the server initiate communication with the client as needed

I fully understand how to implement the former, but being an old-time programmer, I hate polling - the inefficiency just bothers me. To use the second approach, the only method that come to mind is to do some basic socket-level programming.

What magic would allow 2 to work, if the client is not listening all the time, i.e. waiting to be polled ?

srnet,

Fair question! Within the loop() function, one can insert a client.available() to check to see if anything has been sent. You are quite right - this is polling; but at least it is not sending a packet to the server with each iteration of loop().

try MQTT protocol, there is little polling ( little enough to save much bandwidth ) ,

once connection initiated, the client(esp) only fetches data from the 'topic' (created by Pi) it subscribed to

all you do is create numbers of topics matching number of esp's in your system

KASSIMSAMJI - I am somewhat 'familiar' with MQTT as that is one of the protocols behind HomeAssistant and KManOz's code for the SOnOff hardware. My intent was to develop something far simpler and more 'transparent', i.e., one doesn't need to understand/rely on MQTT.

That said, MQTT is a good example because when a command is entered via the HomeAssistant webpage, it is reflected on the device very quickly. This means that a) there is an open communication channel between the device and the server, b) the server pushes to the device, or c) the device is sending a request to the server several times per second. Perhaps understanding the mechanism used by MQTT would provide me with some clues...

KASSIMSAMJI - I pulled out my thumb drive with Kali Linux and my USB WiFi adapter that allows me to use WireShark in promiscuous mode. What I found is that the SOnOff devices, which I have programmed with KManOz's code which uses MQTT is the background, are transmitting packets at an average rate of three times per second. The Raspberry Pi, which is running the server, replies far less frequently - over the 30 second capture period, I only see one 'burst' (about one-half dozen packets) to each of my SOnOff devices.

Based on this, I would say that the MQTT approach is 'active polling', by which I mean that the client is regularly sending packets to the server. ('Active polling' in response to srnet's earlier comment.) What I feel would be more desirable is 'passive polling', whereby the client checks for incoming packets without first having to send one.

So, back to the original question: To accomplish this, must one use socket-level code or is there some other approach that would work?