Is it possible to initialize PROGMEM from a file?

Projects Programming
1

I found this in another Thread and I presume it works

char webpage[] PROGMEM = R"=====(
<html>
<head>
etc
etc
)=====";

To keep things short I have replaced about 130 lines of text with 2 EtCeteras.

It would be nice if it was possible to read in all the text from a separate file - for example something like this

char webpage[] PROGMEM = R"=====(
#include"myHtml.html"
)=====";

Obviously in this case the #include won't be acted on. I'm wondering is there any way to make the compiler read in the text from the file?

...R

2

I'm wondering is there any way to make the compiler read in the text from the file?

The preprocessor will do that. The compiler won't. The IDE only supports some extensions for the type of file to include, though.

3

PaulS:
The preprocessor will do that. The compiler won't. The IDE only supports some extensions for the type of file to include, though.

Can you tell me what I need to do to get the pre-processor to do it?

...R

4

Robin2:
Can you tell me what I need to do to get the pre-processor to do it?

...R

Just like your example, except that the IDE doesn't support the .html extension.

Unless you are thinking about that happening at run time. In which case, you can't do that.

5

I believe that your initial example does NOT work. The ability of quoted strings to have embedded newlines went away some time around gcc version 3. (And painful it was!)

6

It does work. That raw string literal syntax was introduced in C++11.

7

Robin2 - you've said you normally don't use the Arduino IDE, right?
You CAN use avr-objcopy to produce a linkable object file from an arbitrary file, which would be pretty easy to add to makefile-style builds.
The command looks like:

avr-objcopy -I binary -O elf32-avr -B avr --rename-section .data=.progmem.html myhtmlfile.html myhtmlfile.o

linked with the rest of your sketch, you'll find symbols like

_binary__myhtmlfile_html_start
_binary__myhtmlfile_html_end
_binary__myhtmlfile_html_size

Google "objcopy binary blob" and see the avr-objcopy man page for details (it does a lot of things, and the man page isn't particularly wonderful. Good Luck!)

8

That raw string literal syntax was introduced in C++11.

Oh. My eyes just glossed over that new R"delim(text including newlines)delim" syntax !

9

PaulS:
Just like your example, except that the IDE doesn't support the .html extension.

The example won't do what I want. It will just include the text "#include"myHtml.html" in the PROGMEM array.

I want it to open the file and put its contents into the array.

westfw:
Robin2 - you've said you normally don't use the Arduino IDE, right?
You CAN use avr-objcopy to produce a linkable object

For what I have in mind that would be too complicated - I'm hoping for something that would be easily done by newbies.

If this was just for myself I would do it with a Python script.

By the way, thank you all for your inputs.

...R

10

It's taken me a while to get my new laptop to the stage where it would compile an example to demonstrate this - internet was too slow to download another version of the IDE.

Anyway I tried this short program on an Uno

// python-build-start
// action, upload
// board, arduino:avr:uno
// port, /dev/ttyACM0
// ide, 1.8.6
// python-build-end


const char webpage[] PROGMEM = R"=====(
#include "WebPageB.js"
)=====";

char myChar;

void setup() {

  Serial.begin(115200);
  Serial.println("Stuff follows ---");
  for (int k = 0; k < strlen_P(webpage); k++)
  {
    myChar =  pgm_read_byte_near(webpage + k);
    Serial.print(myChar);
  }
  Serial.println();
  Serial.println("--- That's all folks");
}

void loop()
{

}

and as expected this is the output

Stuff follows ---                                                
                                                                 
#include "/media/....../WebPageB.js"

--- That's all folks

(My Python code expands the file name and I have excluded the details)

As you can see it just puts the line of text into the array rather than putting in the contents of the file WebPageB.js

Is it just not possible to get it to read in the file contents?

...R

11

Is it just not possible to get it to read in the file contents?

Apparently, the #include statement between the R"=====( and the )====="; causes it not to be seen by the preprocessor.

Can you put the colored stuff IN the file to be included?

12

PaulS:
Can you put the colored stuff IN the file to be included?

Thanks, and following a lot of Googling I did some experimenting yesterday and with what you call the "coloured stuff" in the text file it does work, in the sense that the contents of the text file gets included without being interpreted as C++ code.

However that is not a practical solution because I want the text file to be a regular HTML file (or Javascript or CSS) that can also be used in a (say) a Python program that does the same thing as an ESP8266 program.

Just to fill in the background, if you look at the code in this Exhibition Thread you will see that all the Browser stuff is included as a char array. But all that stuff just gets in the way of editing the Arduino code and the HTML etc is even harder to edit. In a program for the PC all those things would be in separate files so when you edit them the editor would apply the relevant highlighting and section identification.

I had hoped it might be simple to produce a modified version of the tutorial with the bits in different files. But it does not seem to be possible without introducing techniques that would make things even more complex and off-putting for a newbie.

When creating a web application with Python and Bottle (or Ruby on Rails) you can put an escape sequence into the HTML code which tells the interpreter that the following stuff is to be interpreted as Python (or Ruby) code. I had hoped there might be an equivalent for C++ to tell the compiler that the following stuff is to be included but not treated as C++ code.

Thanks for your interest.

...R

13

Have you considered or tried placing the HTML files on an SD card and coding the ESP8266 to read the HTML into a buffer from the SD card or write the buffer to the SD card as a file. I haven't finished my similar project based on a MSP432, but that is the route I was thinking of going.

14

Perehama:
Have you considered or tried placing the HTML files on an SD card?

No, is the short answer. That would be more complex than what I would like.

Where are you planning to have your SD card?

If you are thinking of using an ESP8266 then I believe it would be possible to upload the supplementary files to SPIFFS but from what I can see that gets rather complicated so I have not explored it.

Whether using an SD card or SPIFFS I think the Arduino code would need to read the file piece by piece and send it to the browser. Of course that must be what happens with Python and Bottle, but there is "black magic" that does it for you - you just need something like in the HTML file.

Now that you have put the idea in my mind I might think a bit more about writing a generic function that would read a SPIFFS file and send it to the browser IF I can see a simple way (simple for newbies) to upload the files to SPIFFS

...R

15

Robin2:
Where are you planning to have your SD card?

SD cards by standard are able to operate in SPI mode, which means you can attach the pins directly to the SPI bus as long as the voltages are the same (3.3 volts last I recall). You need to dedicate a pin to SS.
something I googled for SD and ESP8266 seems to be code to read and write to and from an SD card using the wemos, with pinout and all.

16

Robin2:
Whether using an SD card or SPIFFS I think the Arduino code would need to read the file piece by piece and send it to the browser.

Yes, you would need a char buffer to fill with the file contents. You can limit the file size to the buffer size or to the more complicated code to send the data in packets. I have no experience with the ESP8266, so I don't know what memory constraints it has. I do know if trying to do this on a UNO, it's very memory constrained.

17

Just thinking out loud. Perhaps you could have three include files, containing R"=====(, the html file, and )=====";.

18

PaulS:
Just thinking out loud. Perhaps you could have three include files, containing R"=====(, the html file, and )=====";.

If you mean a file just with R''====( and another with the HTML then I don't think that would work because the HTML would be interpreted as C++.

...R

19

Robin2:
IF I can see a simple way (simple for newbies) to upload the files to SPIFFS

This was my goal with the use of SD. I was hoping anyone with an SD card slot in their computer could upload their web pages, images JavaScript etc. to the SD card following simple naming conventions and organizational structure, then plug it in to the SD slot on my device and presto, apply power and their in business. I haven't fully fleshed out all the details. I moved rather quickly from the ATmega328P to a ATmega1284P, both with ENC28J60 for Ethernet, and now to a MSP432 to meet the device requirements.

20

and another with the HTML then I don't think that would work because the HTML would be interpreted as C++

The preprocessor wouldn't interpret the text as C++. The compiler would see the three file's contents as text, and handle all the stuff between the double quotes after the R just like it would if it were part of the ino or cpp file.