Most Efficient Way of Combining Large Strings for HTML Response

Projects Programming
21

chadwixk:
Thank you, some great alternative suggestions that I'll look into.

But fundamentally, I'm still just trying to better understand how to manipulate strings efficiently in C++. How would you do the long concatenations using char instead of String?

The general point was that if you use static HTML (or other Web objects), maybe stored in flash memory using either SPIFFS methods or PROGMEM, you don't have to do long concatenations. You simply stream the data collections you have to the browser.

If you are, however, joining lots of data together, it is usually best to use automatic variables so the stack memory they have occupied is cleaned up when their containing function terminates. On the ESP8266 you have, by default, 4K bytes of stack available. Strings, or more direct types of memory allocation like malloc/free use the heap and may not be so nicely cleaned up, leading to fragmentation and possible allocation failure.

22

Yes I understand. Some of the HTML is static (i.e. config/settings), but others are dynamic (i.e. a history log table). I think some of these alternatives are likely the direction I'll go.

But I was still just curious how to crack this nut using some variation of concatenation, for future reference and to store in my mental toolbox. For that, it sounds like you are recommending automatic variables, which is a term I've come across in my research, but I'll now explore those more. Today, my HTML variable is a global variable that gets set in each page handler function. I am tracking the free memory now and see it consistently going down with time as the program cycles increase. I'm guessing this is because the HTML history log table gets longer with time.

The learning curve has been a little steep coming from C#, especially string manipulation, which I've always taken for granted in my environments that are not constrained like these hardware devices. So thank you for your constructive help and advice in my journey!

6v6gt:
The general point was that if you use static HTML (or other Web objects), maybe stored in flash memory using either SPIFFS methods or PROGMEM, you don't have to do long concatenations. You simply stream the data collections you have to the browser.

If you are, however, joining lots of data together, it is usually best to use automatic variables so the stack memory they have occupied is cleaned up when their containing function terminates. On the ESP8266 you have, by default, 4K bytes of stack available. Strings, or more direct types of memory allocation like malloc/free use the heap and may not be so nicely cleaned up, leading to fragmentation and possible allocation failure.

23

to build strings with variables in C sprintf is used

24

Using AJAX is a way to deal with static HTML and provide the variable part off band

25

Just don't concatenate anything to the HTML file on the server side.

This solution is completely static. Just host the HTML file and the log.csv file in the SPIFFS, and write your log entries to the log.csv file.

1526984400,Log entry 0
1526984460,Log entry 1
1526984520,Log entry 2
1526984580,Log entry 3

<html>

<div id="log">
  <h2>Log</h2>
</div>

<script type="text/javascript">
  fetch("log.csv")
    .then(response => response.text())
    .then(csvText => csvToTable(csvText))
    .then(htmlTable => addLogTableToPage(htmlTable))
    .catch(e => console.error(e));

  function csvToTable(csvText) {
    csvText = csvText.trim();
    console.log(csvText);
    let lines = csvText.split("\n");
    let table = document.createElement("table");
    lines.forEach(line => {
      let fields = line.split(",");
      let row = document.createElement("tr");
      fields.forEach((field, index) => {
        let cell = document.createElement("td");
        let text = index == 0 ? unixTS2String(field) : field;
        cell.appendChild(document.createTextNode(text));
        row.appendChild(cell);
      });
      table.appendChild(row);
    });
    return table;
  }

  function unixTS2String(str) {
    return new Date(parseInt(str) * 1000).toLocaleString();
  }

  function addLogTableToPage(table) {
    document.getElementById("log").appendChild(table);
  }

  if (!String.prototype.trim) {
    String.prototype.trim = function () {
      return this.replace(/^\s+|\s+$/g, '');
    };
  }
</script>

<style>
  td {
    border: 1px solid #ccc;
    padding: 8px;
  }

  tr:nth-child(even) {
    background-color: #f1f1f1;
  }
</style>

</html>

The result:
fetch_log.png

If you want to use single values/variables, you don't need a file. Just add a handler in the HTTP server, and print the value to the response.

server.on("/value", HTTP_GET, [](){ server.send(200, "text/plain", getValue()); });

int myValue = 42;

const char* getValue() {
  static char buffer[16];
  snprintf(buffer, sizeof(buffer), "%d", myValue);
  return buffer;
}

If you have to send a lot of individual values, group them together, or keep the connection open (e.g. using WebSockets) to minimize TCP and HTTP overhead.

26

somehow two separate topics are discussed here.

27

An excellent guide Wawa. Thank you!

Wawa:
A Beginner's Guide to the ESP8266

28

Wawa:
A Beginner's Guide to the ESP8266

thank PieterP. he is the author.

did you look at my library (StreamLib), examples and project with WebServer? read here