ESP8266 Unsuccessful Sending Webpages (Web Server)

Hello,

I've been struggling with an issue for the past few weeks, and I haven't been able to solve it.
I'm working with the ESP8266 and using the WiFiManager library for my project; I just modified the main page along with the other sub-pages to suit my needs. Initially everything was working great, connecting to the ESP (while an access point) was quick, pages loaded quick, no issues loading/displaying page content.

Eventually I started to really develop the web pages, the actual page content started to grow in length, along with including a fair amount of CSS to make things look nice. But then I started running into a peculiar problem where the pages would start to not transmit properly. I would start to see odd characters at the start of the page, the page layout would break/not fully load, the CSS would not be intact, and various issues of the sort without any noticeable pattern.

I started to research the issue, and came across some topics which discuss how the ESP will not be able to transmit strings larger than 7000 bytes or so due to the limited amount of heap. My pages range from 8000 to 12000 bytes, so I was pretty confident this would be the answer. This is the topic I was referencing server.send() fails to send STRINGS/HTML greater than 6600~ byte long · Issue #3205 · esp8266/Arduino · GitHub .
I followed the advice in the second post, which suggested doing the following for large HTML pages:

String HTML_PART_1(){
  html = "<SOME HTML>";
  html += "much much more HTML";
  return html;
}

String HTML_PART_2(){
  html = "<SOME HTML>";
  html += "much much more HTML";
  return html;
}

server.setContentLength(HTML_PART_1().length() + HTML_PART_2().length());   (or hardcode the length)
server.send(200,"text/html",HTML_PART_1());       <---Initial send includes the header
server.sendContent((HTML_PART_2());                 <--- Any subsequent sends use this function

This solved the issue I was experiencing related to pages seeming "corrupted" and displaying unexpected characters.

However I'm now experiencing other, just as frustrating issues.
After I connect to the ESP while in access point mode, I get occasional to very frequent instances where my mobile phone browser (chrome) will indicate there was an issue loading the page.
I get responses such as:

  • empty response
  • content length mismatch

On the chance that I don't received these error messages the pages will tend to appear to have loaded (confirmed by the displayed URL), however the page content can be empty, incomplete, or even display the previous page's content as if it never was replaced with new HTML.

Here's a topic I found on GitHub which describes a very similar, if not the exact same issue, see third post. Multiple problems with WiFiManager · Issue #170 · tzapu/WiFiManager · GitHub

Has anyone experience similar issues, or have any advice regarding how to handle sending large html pages successfully and consistently? I'd really appreciate any guidance to point me in the right direction.

Here's a bit of the code if anyone could take a look.

void AVV::HandleDiagnostics()
{

 String page = FPSTR(HTTP_HEAD);
 page.replace("{v}", "Diagnostics");
 
 
 page += "<script language='javascript'> first = 0; canset = 1; containerWidth = 0; intervalDuration = 250; drawBar=0;";
 page += "function progressBar(duration, at) { resetStart(at == 0 ? duration : at); containerWidth = document.getElementById('container').offsetWidth; increment = intervalDuration*containerWidth/duration; barWidth = (at == 0 ? (containerWidth * 0.98) : (interval*increment)); drawBar = setInterval('progress('+duration%intervalDuration+')', intervalDuration); } ";
 page += "function progress(lastms) { document.getElementById('bar').style.width=barWidth; if(interval <= 0) { clearInterval(drawBar); document.getElementById('bar').style.width=0; } else { barWidth = interval*increment; } interval--; } ";
 page += "function resetStart(duration) { if(drawBar) clearInterval(drawBar); interval = duration / 250 * 0.98; }";
 page += "function pbclear() { interval = 0;} </script> ";
 
 page += FPSTR(STYLE_DIAG);


 int length = DiagnosticsAJAX().length();
 int lengthb = DiagnosticsContent().length();
 int lengthc = DiagnosticsContentB().length();
 debug("page: " + String(page.length()) + ", " + String(length) + ", " + String(lengthb) + ", " + String(lengthc));
 
 server->setContentLength(page.length() + length + lengthb + lengthc);  
 server->send(200, "text/html", page);
 server->sendContent(DiagnosticsAJAX());
 server->sendContent(DiagnosticsContent());
 server->sendContent(DiagnosticsContentB());
}
String AVV::Diagnostics...()
{
 String page = "";
        page += ...
 return page;
}

A couple of things which may help.

  1. To avoid the continuous String concatenation :
 page += "<script language='javascript'> first = 0; canset = 1; containerWidth = 0; intervalDuration = 250; drawBar=0;";
 page += "function progressBar(duration, at) { resetStart(at == 0 ? duration : at); containerWidth = document.getElementById('container').offsetWidth; increment = intervalDuration*containerWidth/duration; barWidth = (at == 0 ? (containerWidth * 0.98) : (interval*increment)); drawBar = setInterval('progress('+duration%intervalDuration+')', intervalDuration); } ";
 page += "function progress(lastms) { document.getElementById('bar').style.width=barWidth; if(interval <= 0) { clearInterval(drawBar); document.getElementById('bar').style.width=0; } else { barWidth = interval*increment; } interval--; } ";
 page += "function resetStart(duration) { if(drawBar) clearInterval(drawBar); interval = duration / 250 * 0.98; }";
 page += "function pbclear() { interval = 0;} </script> ";
. . . .

You can use 'super quotes' which also make the code much nicer to look at and easier to manage, but beware of too many embedded spaces. E.g. from a page to configure the Web server with initial parameters:

const char MAIN_page[] PROGMEM = R"=====(
<!DOCTYPE html>
<html>
   <head>
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
      <style>
         body {font-family: helvetica,arial,sans-serif;}
         table {border-collapse: collapse; border: 1px solid black;}
         /*table {border: 1px solid black;}*/
         td {padding: 0.25em;}
         .title { font-size: 2em; font-weight: bold;}
         .name {padding: 0.5em;}
         .heading {font-weight: bold; background: #c0c0c0; padding: 0.5em;}
         .update {color: #dd3333; font-size: 0.75em;}
         output {font-size: 0.75em; color: #696969;}
      </style>
   </head>
   <script>
      function myInit() {
      }
      
      function isValidField( fieldName , fieldMaxLength ) {
      //
      // returns boolean
      //  
      var testField = document.getElementById(fieldName) ;
      var isValid = false  ;
      
      if (  testField ) 
      {
        if (  testField.value.search(/\S+/) == -1  ||  testField.value.length >= fieldMaxLength ) {
           testField.style.backgroundColor="LightCoral";
        }
        else {
           testField.style.backgroundColor="transparent";
           isValid = true ;
        }
      }
      return ( isValid ) ;
      }
      
      
      function check() {
      //
      //
      //
      if ( 
         isValidField( "config_ssid" , 32  ) 
         &&  isValidField( "config_psk" , 64  ) 
         &&  isValidField( "config_remoteHost" , 32  ) 
         &&  isValidField( "config_ruleIntlFrom" , 6  ) 
         &&  isValidField( "config_ruleIntlTo" , 6  ) 
         &&  isValidField( "config_ruleLocalFrom" , 6  ) 
         &&  isValidField( "config_ruleLocalTo" , 6  ) 
         &&  isValidField( "config_phpRetrievePath" , 32  ) 
         &&  isValidField( "config_phpStorePath" , 32  ) 
      
       )
      { 
         document.getElementById("callerIdConfigForm").submit(); 
         document.getElementById("updateResponse").value = "configuration sent to server" ;
      } 
      else {
      document.getElementById("updateResponse").value = "error in marked field(s)" ;
      }
      }
      
   </script>
   <body onload="myInit()">
      <form method="post" action="/form"  id="callerIdConfigForm">
      <div class=title>CallerID</div>
      <button type="button" onclick="check()">update</button>
      <p>
         <input class=update size=40 type=text name="updateResponse" id="updateResponse" value="$@updateResponse@$" >  
      <p>
      <div id=divPage1>
         <table>
            <tr>
               <td colspan=2 class=heading>Configuration</td>
            </tr>
            <tr>
               <td>SSID:</td>
               <td><input type="text" id="config_ssid" name="config_ssid" value="$@config_ssid@$" size=30 ></td>
            </tr>
            <tr>
               <td>PSK:</td>
               <td><input type="password" id="config_psk"  name="config_psk"  value="$@config_psk@$" size=30 ></td>
            </tr>
            <tr>
               <td>Host:</td>
               <td><input type="text" id="config_remoteHost"  name="config_remoteHost"  value="$@config_remoteHost@$" size=30 ></td>
            </tr>
            <tr>
               <td>Intl Rule 1:</td>
               <td><input type="text" id="config_ruleIntlFrom"  name="config_ruleIntlFrom"  value="$@config_ruleIntlFrom@$" size=30 ></td>
            </tr>
            <tr>
               <td>Intl Rule 2:</td>
               <td><input type="text" id="config_ruleIntlTo"  name="config_ruleIntlTo"  value="$@config_ruleIntlTo@$" size=30 ></td>
            </tr>
            <tr>
               <td>Local Rule 1:</td>
               <td><input type="text" id="config_ruleLocalFrom"  name="config_ruleLocalFrom"  value="$@config_ruleLocalFrom@$" size=30 ></td>
            </tr>
            <tr>
               <td>Local Rule 2:</td>
               <td><input type="text" id="config_ruleLocalTo"  name="config_ruleLocalTo"  value="$@config_ruleLocalTo@$" size=30 ></td>
            </tr>
            <tr>
               <td>Get path:</td>
               <td><input type="text" id="config_phpRetrievePath"  name="config_phpRetrievePath"  value="$@config_phpRetrievePath@$" size=30 ></td>
            </tr>
            <tr>
               <td>Store path:</td>
               <td><input type="text" id="config_phpStorePath"  name="config_phpStorePath"  value="$@config_phpStorePath@$" size=30 ></td>
            </tr>
         </table>
      </div>
   </body>
</html>

)=====";
  1. Also, instead of serving all the code up directly to the browser client, you can get the client to fetch some itself. I use this technique where I want the web page to be completely static in the sketch. The browser then makes a connection back to the server to fetch the variable data and populate the fields on the web page. For this, you can use the command XMLHttpRequest in javascript.
    // http://stackoverflow.com/questions/3038901/how-to-get-the-response-of-xmlhttprequest
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = function() {
        if (xhr.readyState == XMLHttpRequest.DONE) {
            document.getElementById("jsonIn").value = xhr.responseText  ;
        }
     }
     xhr.open('GET', '/json', false);   // was true
     xhr.send(null);
     . . .

On the ESP8266 the stack, where local variables are held, is limited to 4K .

Did you read "A Beginner's Guide to the EP8266".
The part where compressed files are stored in SPIFFS memory.
Leo..