Web Server Guidance - how to push information to browser

I built a web server that's working very nicely. [It uses a MKR1000 but that isn't really important to my question.]

The user accesses my "web site" (web server), which returns a page to the user's browser. The user can click on buttons on the page that send a request to my web server, and it returns information and/or an updated web page.

Sometimes, my web server has updated information that it wants to send to the user, but since the user hasn't clicked anything on the webpage to cause the browser to do a send, it has no way to update the information on the user's screen.

Many websites do this kind of thing routinely - that is, send unsolicited updates to the user. For example, you might be waiting to chat with support and the website updates your wait time.

Can someone point me to material (references, sketches, etc) that do this kind of thing so I can learn how to add that capability to my web server?

Thanks!

True "push" is done with WebSockets, which are two-way. For that, you'll probably use additional libraries on both the server and client. You can get a lot of the same effect by -- not "pulling" but -- polling at regular intervals. There's even something in between: long polling, where the client makes a request, and the server will hold onto that request and not reply immediately. It waits until there is something to say, or some longer time limit has been reached. That way, it can simulate a push.

But basic polling is easy to implement, and may be sufficient. Modern browser APIs help. Here's an example web page

<!DOCTYPE html>
<title>Polling demo</title>
<h1>Uptime</h1>
<p id="info">one moment...</p>
<script>
setInterval(() => {
  fetch("/latest")
    .then(response => response.json())
    .then(value => {
      document.querySelector("#info").innerText = value;
    })
    .catch(console.error);
}, 3500);
</script>
  • setInterval runs a function periodically
  • that function calls fetch for a path, which returns a Promise of a Response
  • which is handled by then, which calls json on it, which returns a Promise of some JavaScript value that was returned as JSON
  • which is handled by another then, which uses the web page DOM to set the <p> text to that response value
  • any exception that happens along the way will be printed in the Developer Tools Console

On the web server side, you need to distinguish request paths. Some of the more advanced web server libraries have functions for this, but you can also handle it manually. Here's a modified loop of the WiFi101 example WiFiWebServer, along with a new helper function

void contentLengthClose(Client &client, const String &content) {
  client.print("Content-Length: ");
  client.println(content.length() + 2);  // for CR+LF
  client.println("Connection: close");
  client.println();
  client.println(content);
}

void loop() {
  // listen for incoming clients
  WiFiClient client = server.available();
  if (client) {
    Serial.println("new client");
    String startLine;
    String method;
    bool firstLine = true;
    // an HTTP request ends with a blank line
    bool currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // Accumulate first line
        if (firstLine) {
          startLine += c;
        }
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the HTTP request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          if (method == "") {
            client.println("HTTP/1.1 405 Method Not Allowed");
            client.println("Content-Type: text/plain");
            contentLengthClose(client, startLine);
            break;
          }
          if (startLine == "/latest") {
            String info(millis());
            client.println("HTTP/1.1 200 OK");
            client.println("Content-Type: application/json");
            contentLengthClose(client, info);
            break;
          }
          if (startLine != "/") {
            client.println("HTTP/1.1 404 Not Found");
            client.println("Connection: close");
            client.println();
            break;
          }
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          contentLengthClose(client, home_page);
          break;
        }
        if (c == '\n') {
          if (firstLine) {
            firstLine = false;
            // parse e.g.
            //   GET /something HTTP/1.1
            int i = startLine.lastIndexOf("\x20HTTP/");  // space before HTTP
            if (i >= 0) {
              startLine.remove(i);
              i = startLine.indexOf('\x20');
              if (startLine.startsWith("GET\x20")) {
                method = "GET";
                startLine.remove(0, ++i);
                startLine.trim();  // should now be "/something"
              }
            }
          }
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
}

There's code to parse the first line of the request, aka the start-line, primarily to extract the path. But while we're at it, might as well also check the method or verb, to make sure it is one we support. In this case, we only do GET. If the request is something else, we return the appropriate HTTP response, 405

If the request is for that fetch path, then we return some JSON. In this case, it's just a number. If it was a string, you'd have to put quotes around it, but a number requires nothing more.

Modifying the original function, I left (a little bit of) the existing code that returned the HTML page. So first, if the request is for anything other than the root path /, we return 404. If not, then the home page is now saved as String using a raw string

const String home_page = R"~~~(<!DOCTYPE html>
<title>Polling demo</title>
<h1>Uptime</h1>
<p id="info">one moment...</p>
<script>
setInterval(() => {
  fetch("/latest")
    .then(response => response.json())
    .then(value => {
      document.querySelector("#info").innerText = value;
    })
    .catch(console.error);
}, 3500);
</script>
)~~~";

If you've been following along, you might think, "In this demo, the value is a single number, which is valid JSON. For example, 1234. This goes "over the wire" as four bytes, also 1234, which is then parsed as JSON, returning the number. That means in the function above, value is a number. Can you assign that to innerText? Yes, JavaScript will "helpfully" do those types of conversions for you automatically.

Thank you for the detailed response. I'm going to dig into it and do testing with something simple to try to understand everything.

You brought up Long polling, which seems to answer my need if I understand it correct. What I'm doing now, when I know the server has an update that will be ready in seconds is add the following to the webserver's response:

<meta http-equiv=\"refresh\" content=\"2\">"

But often the update I would like to send happens several minutes later (if at all), so long polling does seem like the answer. Two followup questions:

1-Assuming the user leaves the webpage up in the browser, how long can long be before a response is sent?
2-If the user closes the browser window, does the webserver response just get dropped on the floor (which is fine), or does it cause some problem?

I'd like to learn all about this, but the HTML and JSON documents I've found kind of explain individual items, rather than a context of how they fit into real world needs and usage.

Thank you again.

Seems like the browser will wait for a long time. The internet claims that it does not work as well on Firefox, but I can't test that easily at the moment. Here's a new helper function, and a slight modification to the top of the loop

bool checkLongPoll(Client &client, bool start = false) {
  static auto pollStart = millis();
  if (start) {
    pollStart = millis();
    return true;
  }
  if (!client.connected()) {
    Serial.println("client gave up");
    return false;
  }
  if (Serial.available()) {
    String input = Serial.readStringUntil('\n');
    String value(input.toInt());
    Serial.print("input number: ");
    Serial.println(value);
    client.println("HTTP/1.1 200 OK");
    client.println("Content-Type: application/json");
    contentLengthClose(client, value);
    return false;
  }
  if (millis() - pollStart > 99'000) {  // can try removing this entirely
    client.stop();
    Serial.println("stopped long poll");
    return false;
  }
  return true;
}

void loop() {
  static WiFiClient client;
  static bool longPoll;
  if (longPoll) {
    longPoll = checkLongPoll(client);
    if (!longPoll) {
      delay(1);  // give a chance for everything to settle after stopping
    }
    return;  // even after stopping, let main loop do its business
  }
  client = server.available();
  if (client) {

It has a hard stop at an arbitrary 99 seconds; try altering or removing it. The modification to the /latest handler:

          if (startLine == "/latest") {
            longPoll = checkLongPoll(client, true);
            Serial.println("start long poll; enter number in Serial Monitor with New Line");
            return;
          }

and the updated web page

<!DOCTYPE html>
<title>Long polling demo</title>
<h1>The number that was input</h1>
<p id="info">awaiting "push"</p>
<script>
let backoff = 0;
async function longPoll() {
  try {
    const response = await fetch("/latest");
    backoff = 0;  // fetch didn't fail; don't increase backoff
    const value = await response.json();
    document.querySelector("#info").innerText = value;
  } catch (x) {
    console.warn(x);  // to distinguish from normal errors
  }
  if (backoff) {
    backoff *= 3;
  } else {
    backoff = 250;
  }
  setTimeout(longPoll, backoff);
}
longPoll();
</script>

The server function checks whether the connection is dropped by the client. And there's always some chance you'll send something as the client disappears. Not a problem.

All this assumes the plain old WiFiServer doesn't handle simultaneous connections, as a more capable web server would. If it's just a single connection, when handling a long poll, it can't respond to another client, or even the same client if the connection is not cleared correctly.