How to create dynamic webpages on ESP8266 which use least RAM & Flash

Projects Programming
1

Ok, so even though i am expecting a rather complex response from people i decided to ask this question regardless.

I use ESP8266's a lot, and generally to create webpages of all sorts.

Now at some point RAM usage becomes an issue in the way that i was doing it.
I create a local String and add whatever part of the webpage i want, and send it either in Chunks (Chunked content) or as 1 single stream.

For different pages there are sections that are always the same or very similar, so to prevent lots of mistakes from typos, i started out with making things like the Header, the bodystyle etc a const char * that i define once and use in many places. This of course goes at the expense of RAM, so i though if i make them
static const char characters[] PROGMEM "896*^(";
and call the using

s += FPSTR(characters);

they are placed in PROGMEM, and they are only added to the 'String' locally, so they do not impact global RAM usage, nor do they impact Flash memory much (i am adding to the String on the fly with the same section of characters that is being loaded from PROGMEM) and i do prevent the risk of excessive typos, which during development of HTML in particular makes quite a lot of difference, since the compiler does not check for mistakes.

Now i found that i am impacting RAM ! I couldn't work out where all of my global RAM usage was coming from, until i started commenting out some of the compiled pages, and all of a sudden the RAM usage dropped.

So i figured i am doing something wrong, and it is time to fix that, and where better to ask than here.
an example of what i have doing so far

static const char pageheader[] PROGMEM = "<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN''http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>";
static const char htmlhead[] PROGMEM = "<html><head><title>DmagiX LED Controllers</title><meta http-equiv='Content-Type' content='text/html;charset=utf-8' ></head>";
static const char headmetaopen[] PROGMEM = "<html><head><title>DmagiX LED Controllers</title><meta http-equiv='Content-Type' content='text/html;charset=utf-8'";
static const char headmetaclose[] PROGMEM = " ></head>";
static const char bodystyle[] PROGMEM = "<body style='color: wheat; background-color: DarkSlateBlue; font-size: 12pt; font-family: sans-serif;'>";
static const char htmlclose[] PROGMEM = "</body></html>";


void MainPage() {
  server.setContentLength(CONTENT_LENGTH_UNKNOWN);
  String s;
  s += FPSTR(pageheader);
  s += FPSTR(htmlhead);
  s += FPSTR(bodystyle);

  s += "<h1>Welcome to DmagiX Wifi !</h1><p>From here you can control your DmagiX Controller. DmagiX Patterns consist of 2 layerss, with their own ";
  s += "RGB color, and a few global parameters. There are also 30 presets available for quick recall, and a random mode that can have a timed trigger.";
  s += "</p>";

  s += "<form action='";
  s += webIP;
  s += "' method='get' name='todmagix'><input type='hidden' name='page' value='dmagix'>";
  s += "<input type='hidden' name='request' value='dmagix'>";
  s += "<input type='submit' value='    DmagiX     '></form><br>";

  s += "<p>General Settings can be modified at any time and of course be bookmarked for later use. The Settings that were present ";
  s += "at login will be kept for easy retrieval and the option to actually store the new settings on the EEPROM is separate from applying them. ";
  s += "Settings include the number of LED's the RGB sequence, Random-mode trigger, as well as settings relating to DMX. </p>";
  server.send(200, "text/html", s);
  s = "";

  s += "<form action='";
  s += webIP;
  s += "' method='get' name='tosettings'><input type='hidden' name='page' value='settings'>";
  s += "<input type='submit' value='    Settings    '></form><br>";

  s += "<p>A DmagiX unit starts up as an Access Point that you can log onto, but it can connect to a WiFi network just as easily, and you can store";
  s += "a network that you have connected to, and it will look for stored networks to connect to at startup. A dynamic IP address will be assigned ";
  s += "which is found on the WiFi page after connection, and a mDNS-lookup name can set and stored. Also you can change the Access Point name into ";
  s += "a more meaningful one and you can disable the password.</P>";

  s += "<form action='";
  s += webIP;
  s += "' method='get' name='wifidata'>";
  s += "<input type='hidden' name='page' value='wifi'>";
  s += "<input type='submit' value='       WIFI      '></form><br>";


  s += "<p>User Accounts can be set up to protect certain areas and settings. A device specific Admin password allows you to set up a primary ";
  s += "user account which can modify all other accunts and any data on the controller.<br> This is unit : <b>";
  s += DefaultAPname();
  s += "</b></p>";
  server.sendContent(s);
  s = "";

  s += "<form action='";
  s += webIP;
  s += "' method='get' name='userdata'>";
  s += "<input type='hidden' name='page' value='user'>";
  s += "<font size='+2'>";
  s += "<input type='submit' value='  Accounts  '></font></form><br>";

  s += "<form action='";
  s += webIP;
  s += "' method='get' name='changeuser'>";
  s += "<input type='hidden' name='page' value='maincu'>";
  s += "<input type='submit' value='Change User'></form><br>";

  if (CheckAuth() & B100) {   // so the backdoor also reveals the admin password
    s += "<b>'admin' password : ";
    s += AdminPW();
    s += "</b><br>";
    s += TimeSwitchedOn();
    s += "<br>User accounts are turned ";
    if (useuser) s += "on";
    else s += "off";
    s += ".<br>Guest account is turned ";
    if (guestaccount) s += "on";
    else s += "off";
    s += ".<br><br>";


    //s += TestFrameStoreTime();

    s += "<br>";

    s += "<form action='";
    s += webIP;
    s += "' method='get' name='reboot'>";
    s += "<input type='hidden' name='page' value='reboot'>";
    s += "<input type='submit' value='Reboot Unit'></form><br>";
  }
  s += FPSTR(htmlclose);
  yield();
  server.sendContent(s);
  server.sendContent("");
}

It seems that these global PROGMEM c-strings are not put in PROGMEM, and i fact i suspect that there is even a copy stored for each time i use them in a page.

This example page does not contain much dynamic content, but most of the other pages contain a lot of content that is added on the fly so storage in SPIFFS, which should reduce the RAM usage, may not be so practical. (sure it can be done, but i'd be reading from several different files for every page thaat is being displayed, so i would prefer to avoid that option)

2

I was also wondering if this may be caused by all of this being in the callback function of the ESP8266webserver ?

3

This probably is not the exact answer to your question but there is a problem with Strings when memory is running low. Please see: Detecting errors in String operations · Issue #186 · arduino/ArduinoCore-API · GitHub
One way to get around it is using concat String member function instead of += operator. The benefit is that you can actually check the success of concatenation:
concat() - Arduino Reference

4

When you add the constant Strings to the web page they are copied to ram as part of the String. String operations will deallocate and reallocate memory as they grow.

There are a couple of solutions. One use character arrays (lower case string) and concatenate. This requires that you know the web page length so you can allocate enough space.

If you want to use Strings (upper case) allocate the space ahead of time with the .reserve() method to minimize memory reallocation problems. Obviously you again have to know the max web page size ahead of time. This assumes the compiler/libraries for the particular processor handle these options correctly.

5

when you concatenate Strings a lot (like you do), you should reserve memory with s.reserve(4200) to avoid memory fragmentation.

6

have you considered ESPAsyncWebServer

7

Strangely enough, and i never actually used that before, it seems that if you add constants or literals to Strings, All of these Strings are allocated RAM on the Stack, globally ! And using the F() macro tells them to be put in PROGMEM.
I always figured that if i add a literal to a String, that the literal would be put in flash, or at least would be put on the stack or heap locally, but this somehow turned out not to be the case.

  s += F("<h1>Welcome to DmagiX Wifi !</h1><p>From here you can control your DmagiX Controller. DmagiX Patterns consist of 2 layerss, with their own ");
  s += F("RGB color, and a few global parameters. There are also 30 presets available for quick recall, and a random mode that can have a timed trigger.");
  s += F("</p>");

  s += F("<form action='");
  s += webIP;
  s += F("' method='get' name='todmagix'><input type='hidden' name='page' value='dmagix'>");
  s += F("<input type='hidden' name='request' value='dmagix'>");
  s += F("<input type='submit' value='    DmagiX     '></form><br>");

I still have a hard time believing that this is actually true, because somehow it seems to me that to put all of that in PROGMEM is what almost everybody would want. I always thought that it might make some difference in the local stack or heap use, but apparently the default behavior is that all of those literals are first put into RAM when the program starts running, and are then retrieved from there when they are needed.
Anyway, using that F() macro freed up the memory for me, and with the help of Notepad++ and the find/replace it wasn't an excessive amount of work.

Memory fragmentation is of course another issue

Well yes, but for a local String that does not really matter to much, since whatever is placed on the heap is cleaned up when the function is exited and all those variable go out of scope.

If i add the returning String from a function to a local String inside the function it is calling

String s;
s += server.arg("myArg");

There is a String created to contain "myArg", which is used to compare within server.arg(), the resulting is added to 's' . Chances are that unless 's' has space to grow (either it has been allocated size as a result of reserve() or it has been bigger before, the resulting String is added in place and all the other Strings are destroyed there after (except for the String holding all the arguments, which is only destroyed once the callback function is exited).

If the String does get moved, the area that it occupied before gets freed, and may be used again by other String that fit in it. All of this until the function in which 's' was declared is exited and all of that space is freed up again.

That means all of these local Strings are destroyed by the end of loop() for sure and only the global ones remain in memory, so it makes sense to use .reserve() to declare space for these and make sure that you don't exceed that limit or you may create persistent memory fragments.

btw.

String thestring = "12345678";
thestring = "";

comes down to the same on the heap as

String thestring = "";
thestring.reserve(8);

what i missed, was that

String s;
s += server.arg("myArg");

here the literal "myArg" gets allocated stack space at compile time as a literal, so that it can quickly be put on the heap as a String at runtime. and that if i want it to be retrieved directly from PROGMEM, if have to tell the compiler to do so. (this is of course slower) And i assume that this goes for all variables that are than 32-bit and require a pointer. I mean there are assembler instructions that put a 32 bit variable straight into memory from a value, but not for variables that are bigger. There are instructions to move blocks of data from RAM to RAM, but for moving data from PROGMEM into RAM a different process is needed.

So anyway

String s;
s += server.arg(F("myArg"));

This uses up less global stack space, since the variable is not retrieved from progmem and put on the stack at the start of runtime, but retrieved from PROGMEM during runtime when the instruction is processed.

I am going to mark this myself, as a solution, and even though i have seen mentions of this F() macro many times before, i finally seem to understand.

That said, i am not 100% sure of some details on the matter.

  • does the same thing apply to char* & arrays, ? I suspect so, since it seems to apply to the literal used in the assignment, not the variable the value is assigned to.
  • are there differences between 8-bit & 32-bit MCU's on the matter of 32-bit variables ? or do the 8-bit MCU's have the same 'put this 32 bit variable in these 4 bytes at runtime instruction' (i suspect so)

In the end it comes down to the trade of between speed and memory usage i guess. PROGMEM retrieval is slower and if it is all done at the beginning of the program it will speed up the flow of processing.

The eternal programmers choice between the usage of resources, that on an ESP i never really paid attention to on an ESP since memory was so abundant, and on an AVR because memory was so scarce that using any kind of literal Strings (or c-strings for that matter) was minimized anyway.

Thank you all, i have solved my issue with the help of people that read and responded to this thread, and if there are any clarifications i would always be interested in reading them.

8

reduce concat:

  s += F("<h1>Welcome to DmagiX Wifi !</h1><p>From here you can control your DmagiX Controller. DmagiX Patterns consist of 2 layerss, with their own "
         "RGB color, and a few global parameters. There are also 30 presets available for quick recall, and a random mode that can have a timed trigger."
         "</p>"
         "<form action='");
9

Yeah sure that may help locally, but globally has no effect.

10

It is also possible to send HTTP replies in chunks, of, say 1500 bytes, while constructing it, without having to construct the whole HTTP reply in a single String and then sending it all at once. I believe this would be preferred option.

11

If you look at my initial post, you see that i am already doing that.

  server.setContentLength(CONTENT_LENGTH_UNKNOWN);
  String s;
(-----)
  server.send(200, "text/html", s);
  s = "";
(-----)
  server.sendContent(s);
  s = "";
(-----)
  server.sendContent(s);
  server.sendContent("");

But yes that helps of course, the size of 's' doesn't become the size of the full page, and the memory allocated size will not exceed the maximum size of the chunk.

12

Why not enter the 21st Century, and use the AsyncWebServer, with all the content stored in files in a FLASH filesystem. It GREATLY reduces the amount of code you need to write, and makes "dynamic" web content as simple as it is on a "real" computer, like a PC. Building String responses is pretty much a stone-age technique at this point. Once you use the AsyncWebServer, you will NEVER go back.