Most Efficient Way of Combining Large Strings for HTML Response

I’m a C# web apps developer dabbling in building IOT devices. My first attempt is a direct Wifi enabled thermostat that you can control via a web interface for my popup camper so I can control it at 3am when I’m inevitably roasting or freezing without getting out of my sleeping bag and crawling over my wife :slight_smile:

I have a basic working code and wired up hardware circuit board using a NODEMCU 12E ESP8266 device and a DHT22. However, it is not reliable. It seems after a certain number of operations, it freezes up. After researching, I’m guessing it has to do with the fact that I am using the evil String class do handle my large HTML building strings.

I’ve literally been researching the best way to do this for a couple of months (this is a hobby, but taking the camper out next weekend, so I now need to finish it up). I see a ton of posts about not using the String class as it is a memory hog…got it. But I have not found straight forward answers to concatenate large strings like what I’m trying to do.

Below is a snippet of my sketch that builds the HTML for the most problematic web page in that it results in the most string concatenations and the longest string. Upon initial run it is 2k + in length. But note it also calls a looping routine in readHistoryDataFile() which is more of the same concatenation of the webString variable.

What is an efficient way to do this looping string building?

Thanks in Advance!

void setHTMLHistory()
{
  String webString = "";   // String to build and return HTML response
  webString = "<html>\n";
  webString += "<head>\n";

   ////////////SCRIPTS
   webString += "<script>\n";  
   webString += "var hostName = window.location.hostname;\n";
   webString += "function convertToDateTime(minAgo){ \n";
   webString += "var amPM = 'a'; \n";
   webString += "var dt = new Date(); \n";
   webString += "var currentMsec = dt.getTime(); \n";
   webString += "dt.setTime(currentMsec - (minAgo * 60000)); \n";
   webString += "var month = dt.getMonth() + 1; \n";
   webString += "var hours = dt.getHours() + 0; \n";  
   webString += "if (hours > 12) { hours = hours - 12;  amPM = 'p';} else { amPM = 'a'; } \n";
   webString += "var mins = dt.getMinutes() + 1; \n";
   webString += "if (mins < 10) { mins = '0' + mins; } \n";
   webString += "var dteString = month + '/' + dt.getDate() + '/' + dt.getFullYear() + ' ' + hours + ':' + mins + amPM; \n";
   webString += "document.write(dteString); \n";
   webString += "} \n";
   webString += "</script>\n";  
  
  ///////////// styles
  webString += webStyles;
  webString += "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n";  
  webString += "</head>\n";
  
  ///////////BODY
  webString += "<body style='font-family:Arial;'>\n";
  
  webString += "<h1>History</h1>\n";
  webString += "<div style=\"color:red;\">" + webMessage + "</div>\n";
  
  ///////////Menu
  webString += "<button onclick=\"window.location.href='/'\">Home</button>\n";
  webString += "<button onclick=\"window.location.href='/history'\">Refresh</button><br \\>\n";
  webString += "<hr>\n"; 
  ///////////SETTINGS
  webString += "<h4>History Settings</h4>\n";

  webString += "<form action=\"/updatehistorysettings\" method=\"POST\">\n";

  webString += "<table>\n";
  
  webString += "<tr>";
  webString += "<td>Check Temp Every (mins):</td>";
  webString += "<td><input type='text' value='" + String((long)sampleInterval / 60000) + "' name='sRate' maxlength='3' size='2' style=\"width: 50px;\"></td>";
  webString += "</tr>\n";

  webString += "<tr>";
  webString += "<td>Maximum History Lines:</td>";
  webString += "<td><input type='text' value='" + String((long)maxFileData) + "' name='maxData' maxlength='3' size='2' style=\"width: 50px;\"></td>";
  webString += "</tr>\n";
  
  webString += "<tr>";
  webString += "<td>Current Data Lines:</td><td>" + String((int)dataLines) + "</td>";
  webString += "</tr>\n";
  
  webString += "</table>\n";

  webString += "<input type=\"submit\" value=\"Update\" />\n";
  webString += "</form>\n";

  ///////////DATA
  webString += "<h4>History</h4>\n";
  webString += [b]readHistoryDataFile()[/b] + "\n";

  webString += "
\n";
  webString += "<form action=\"/clearhistory\">\n";
  webString += "<input type=\"submit\" value=\"Clear History\" />\n";
  webString += "</form>\n";

 ///////////////////////SPIFFS STATUS
  webString += "<h4>Disk Usage</h4>\n";
  webString += "<table>\n";
  
  webString += "<tr>";
  webString += "<td>Total KB:</td>";
  webString += "<td>" + String(fileTotalKB) + " KB</td>";
  webString += "</tr>\n";

  webString += "<tr>";
  webString += "<td>Used KB:</td>";
  webString += "<td>" + String(fileUsedKB) + " KB</td>";
  webString += "</tr>\n";

  webString += "<tr>";
  webString += "<td>Remaining KB:</td>";
  webString += "<td>" + String(fileTotalKB - fileUsedKB) + " KB</td>";
  webString += "</tr>\n";
  
  webString += "</table>\n";
  
  webString += "
\n";
  webString += "<form action=\"/formatdisk\">\n";
  webString += "<input type=\"submit\" value=\"Format Disk\" />\n";
  webString += "</form>\n";
  
  webString += "</body></html>\n";

  Serial.printf("    webString Length: %lu\n", webString.length());
}

I've literally been researching the best way to do this for a couple of months

It should not take more than a couple of days of study and focused effort to remove all the failure-causing String crap, and replace it with C-strings (character arrays).

Concatenating C-strings is trivial, just use the strcat() function.

Or use snprintf() to generate the entire string at once, from the base variables and a well thought out format C-string.

If you do this correctly, you can have confidence that your program will run for a long time without the unintended crashes caused by String object manipulations.

Serial.print ( F( "This is a very" " "
"very, very, very" " "
"very, VERY" " "
"long string, perfectly concatenated" " "
"by the C compiler"
));

use my StreamLib. search in Library Manager. it has CStringBuilder, BufferedPrint, ChunkedPrint for html chunked transfer and printf support

I think whole code must be rewritten if its written like this function.
There is no need for ‘\n’ between html tags. Why to waste space even there too?

This whole thing is to calculate webstring.length?

As to avoid confusion as much as possible and to get to the root of what I’m hoping to see is this. Let me start with a very simple snippet of code, and if you are willing, provide edits to it.

This is essentially how I have it today. How would you change it?

string returnHTML()
{

     String strHTML = '';
      
      int j =0;
      
      while (j < 20000) {
        strHTML += millis() + ": " + j.toString() + "This is a test.";
        j++;
      }

      return strHTML;
}

Surely you are joking.

You offer function code that (if it worked at all) would create a String that contains about 20x20000 characters on an Arduino, when you have been advised not to use Strings at all?

Nope, no joking. It was pseudo code to convey a concept of what I’m looking for and asking the experts to fix it. Are you able to fix it?

Here, I’ll simplify it more and made sure it compiles.

void setup() {
  // put your setup code here, to run once:
Serial.begin(9600);
}

void loop() {
  // put your main code here, to run repeatedly:
  String test = returnHTML();
  Serial.println(test);
  delay(500);
}
  
String returnHTML()
{
     String strHTML = "";     
     int j = 0;
      
     while (j < 20) {
        strHTML += "This is a test.\n";
        j++;
      }
      return strHTML;
}

jremington:
Surely you are joking.

You offer function code that (if it worked at all) would create a String that contains about 20x20000 characters on an Arduino, when you have been advised not to use Strings at all?

Why build a full string? You can print the individual parts piece by piece; they end up in a buffer before being send.

Serial.print("Hello ");
Serial.print("World!");

has the same effect as

Serial.print("Hello World!");

sterretje:
Why build a full string? You can print the individual parts piece by piece; they end up in a buffer before being send.

Serial.print("Hello ");

Serial.print("World!");



has the same effect as


Serial.print("Hello World!");

true on Serial, not on network Client implementations where it could be send over network in two parts

for a big http request use chunked transfer. see example in StreamLib, and read the More info link for this lib in Library Manager

You can also use “super quotes” (C++11 raw string literal) which handles embedded quotes in the text without having to litter it with escape characters e.g. :

const char MAIN_page[] PROGMEM = R"=====(

<!DOCTYPE html>
<html>
<head>
<style>
 body {font-family: helvetica,arial,sans-serif;}
 </style>
</head>

<body onload="myInit()">

<div class=title>NTP Clock</div>

<button id="btn1" onclick="goPage(1)">1</button>
<button id="btn2" onclick="goPage(2)">2</button>
<button id="btn3" onclick="goPage(3)">3</button>
<button id="btn4" onclick="goPage(4)">4</button>
<button onclick="check()">update</button>
<p>

<script>
function myInit() {
var pageNo;
var xhr = new XMLHttpRequest();
// . . .
}
</script>

</body>
</html>

)=====";

There are some tricks to add variable information into the html:

  1. embed markers in your static html e.g. TEMP and use a string replace function to put in the current values, then send the page to the browser.
    or
  2. embed java script code in your html which uses the XMLHttpRequest() method to make a return trip to the server to collect all the variables (maybe in json format) and uses the document object model functions within the browser to add in all your variables in to the html page.

I have some (not beautiful) examples of both.

If you are building configuration type pages with lots of fields, but to be displayed on a small screen, another trick is to do that with a “single page application”, that is send one physical page to the browser which is divided up into a number of logical pages and your page forward/backward button simply hides everything but the current logical page. This means only one trip to the server and the user can change “page” after adding data and without having to do a submit() each time. There may be other ways of achieving the same thing.

You can also save .html, .css, .ico etc. files directly to SPIFFS with the DATA function of the IDE.
Leo..

Using Arduino to do the heavy string manipulation is a pain. The best solution I've found is to put all thr files in SPIFFS, like Wawa suggested, and then use JavaScript to build the webpage. All files on the Arduino are static: you can stream them without changing them.
To get variables in your JavaScript, just send another request to the Arduino (either using HTTP if it's a one-time action, or using WebSockets if you want the values to be updated in real-time without refreshing the page). The second request can just send the values in any format you like. If you need just a few values, just send them as plain text with some separator (us snprintf), otherwise, you could use JSON for example.
I see you're trying to add a data file into the HTML: just use JavaScript to fetch that file using a second request.

Pieter

SPIFFS and AJAX is usually the way to go if you want good performance

PieterP:
Using Arduino to do the heavy string manipulation is a pain. The best solution I've found is to put all thr files in SPIFFS, like Wawa suggested, and then use JavaScript to build the webpage. All files on the Arduino are static: you can stream them without changing them.
To get variables in your JavaScript, just send another request to the Arduino (either using HTTP if it's a one-time action, or using WebSockets if you want the values to be updated in real-time without refreshing the page). The second request can just send the values in any format you like. If you need just a few values, just send them as plain text with some separator (us snprintf), otherwise, you could use JSON for example.
I see you're trying to add a data file into the HTML: just use JavaScript to fetch that file using a second request.

WebServer in my project is an example of this approach. here are the static files. and it is an example of StreamLib usage.

In the bigger picture of this code, this function simply builds the HTML and passes it back to the http client response handler to send it to the requesting client. So it needs to be in a variable to pass around, not serial printed.

sterretje:
Why build a full string? You can print the individual parts piece by piece; they end up in a buffer before being send.

Serial.print("Hello ");

Serial.print("World!");



has the same effect as


Serial.print("Hello World!");

Thank you, some great alternative suggestions that I’ll look into.

But fundamentally, I’m still just trying to better understand how to manipulate strings efficiently in C++. How would you do the long concatenations using char instead of String?

6v6gt:
You can also use “super quotes” (C++11 raw string literal) which handles embedded quotes in the text without having to litter it with escape characters e.g. :

const char MAIN_page[] PROGMEM = R"=====(
body {font-family: helvetica,arial,sans-serif;}
NTP Clock

1
2
3
4
update

)=====";





There are some tricks to add variable information into the html:

1. embed markers in your static html e.g. ***TEMP*** and use a string replace function to put in the current values, then send the page to the browser.
or
2. embed java script code in your html which uses the XMLHttpRequest() method to make a return trip to the server to collect all the variables (maybe in json format) and uses the document object model functions within the browser to add in all your variables in to the html page.

I have some (not beautiful) examples of both.

If you are building configuration type pages with lots of fields, but to be displayed on a small screen, another trick is to do that with a "single page application", that is send one physical page to the browser which is divided up into a number of logical pages and your page forward/backward button simply hides everything but the current logical page. This means only one trip to the server and the user can change "page" after adding data and without having to do a submit() each time. There may be other ways of achieving the same thing.

This will work for some elements, thank you. And you are refering to something like this, correct? http://www.instructables.com/id/Using-ESP8266-SPIFFS/

Wawa:
You can also save .html, .css, .ico etc. files directly to SPIFFS with the DATA function of the IDE.
Leo…

Wawa:
You can also save .html, .css, .ico etc. files directly to SPIFFS with the DATA function of the IDE.
Leo…

https://tttapa.github.io/ESP8266/Chap01%20-%20ESP8266.html