Fixed Address Variables?

Hello!

I am trying to make a system that would allow me to load a pre-compiled C++ program off an SD card and read it back as bytecode (on a processor that supports this, of course). However, there's a problem: I need to access variables from the parent program (Such as an SD class, a tft screen class, etc.).

This could easily be done if these variables could be placed at designated spots in memory.
Is this possible?

Thanks!

P.S. I would not place all variables in dedicated spots, only a couple (specifically, the ones that need to be accessed by a child process)

"read it back as bytecode"? You lost me.

In the arduino IDE, there is an option to export a compiled version of your code as a binary file (what I was calling "bytecode"). Reading this into an array, then calling this array as a function, would run the compiled code.

Sorry if I was unclear!

Nope, that's not going to work. You can't call an array as a function.

Pete

void (*foo)(void)=reinterpret_cast<void (*)()>(&myArray);
foo();

AVRs do not allow running code that resides in RAM, so what you want to do is impossible.

Regards,
Ray L.

Fuzzyzilla:
on a processor that supports this, of course

This is on the ESP32, which uses a shared address space and can easily read code from RAM.

This could easily be done if these variables could be placed at designated spots in memory.

yeah, it would look like the PORTx "variables defined for AVR":

#define VAR1 *((int*)0x20001024)

Or you could do something with shared memory segments, positioned with linker options.
This isn't going to be easy; the "byte code" you think you're saving will tend to have a lot of extra startup code, and/or not be fully linked, and/or not be relocatable.

Fuzzyzilla:
This is on the ESP32, which uses a shared address space and can easily read code from RAM.

That's the kind of detail that should be in the first post.

Based on my brief googling around, GCC (this appears to be the compiler ESP uses) doesn't have a way to do this in the code. It doesn't support the at() attribute. It needs to be done with a linker script, which I don't know how to get to work with Arduino IDE. You would probably have any easier time using their full IDE.

There's a very high chance (certainty, I think) that you will need a special setup to compile the code you load from the SD card too. Normal compilation will be based on the assumption that your code is loaded into a certain address. When you load it into RAM like this, that assumption is violated and any instruction that uses direct addressing will be screwed up. A RAM and program buffer will need to be allocated in the parent sketch at fixed locations and of fixed size, and the child program will have to be compiled to run out of those locations.

westfw:
yeah, it would look like the PORTx "variables defined for AVR":

#define VAR1 *((int*)0x20001024)

That wont' work. The peripheral registers already exist, the compiler just needs to know where to access them.

For placing variables in SRAM, it's not enough to just create a fixed-address pointer. You also need to have the memory allocated so that the compiler doesn't attempt to put anything else there.

westfw:
This isn't going to be easy; the "byte code" you think you're saving will tend to have a lot of extra startup code, and/or not be fully linked, and/or not be relocatable.

Jiggy-Ninja:
There's a very high chance (certainty, I think) that you will need a special setup to compile the code you load from the SD card too. Normal compilation will be based on the assumption that your code is loaded into a certain address.

This might be a lost cause o_O. Thank you both, though!

As sort of an afterthought, is there any better ways you can think of?
I've already written an interpreter for Forth in an attempt to do this with an interpreted language, but it's a long, low-level language that seems a bit un-fit for the job.

For placing variables in SRAM, it's not enough to just create a fixed-address pointer. You also need to have the memory allocated so that the compiler doesn't attempt to put anything else there.

Well, yeah. But that should be comparatively easy to achieve; all you have to do is modify the linker script to exclude some of the known RAM - much simpler than managing the build process with new sections...

As sort of an afterthought, is there any better ways you can think of?

An interpreter is a good idea. There are subsets of Forth that would be much smaller than a full implementation. Have you seen "Bitlash"? IICR, it can chain source from an SD file (I'm not sure whether it's been ported to ESP, though.) What about python - does ESP run a version of Python?

Here's how that used to be handled in the distant past with simple processors, by creating a "page relocatable file":

For the code that has to be loaded from SD and run, compile it twice, with two different starting addresses 256 bytes apart. Then compare the two binaries. You'll find bytes that differ by 1 between the two files. Note the locations of those bytes, and build a bit map to tell you where there are, as offsets from the start of the image. Discard the image with the 256 byte offset, and put the one with the 0 byte offset on the SD card, along with the bitmap. When you load the image, load it starting at a 256 byte "page", and use the bitmap to adjust the value of each of those bytes to correct for the 256 byte "page corresponding to the actual load address.
It's a PITA, but does work with many processor architectures. With many (perhaps most) modern CPUs, there will very likely be aspects of the target CPU instruction encoding that make it more complicated, possibly by a lot.

Regards,
Ray L.

Fuzzyzilla:
This might be a lost cause o_O. Thank you both, though!

I'm not saying it's a lost cause. Far from it, this kind of thing is what von Neumann architectures are built for.

I'm just saying that's it's very much "off the beaten path" for an Arduino project, and quite advanced. It is possible to fix variables at a set address, but GCC requires it to be done in a linker script and not the source code. So study how that part of the GCC toolchain works. Study the Arduino IDE Third party Hardware Specification. Study the platforms.txt file that comes with the ESP32 core, and see if there's any place you can add your linker script. Try to make your own hardware folder entry that uses the ESP32 files, but modifies the build process to include the linker scripts you want.

westfw:
Well, yeah. But that should be comparatively easy to achieve; all you have to do is modify the linker script to exclude some of the known RAM - much simpler than managing the build process with new sections...

I'm confused. You claim that modifying the linker script to manage the build process to exclude a new section of RAM is easier than...managing the build process with new sections of RAM? What am I missing?

RayLivingston:
Here's how that used to be handled in the distant past with simple processors, by creating a "page relocatable file":

For the code that has to be loaded from SD and run, compile it twice, with two different starting addresses 256 bytes apart. Then compare the two binaries. You'll find bytes that differ by 1 between the two files. Note the locations of those bytes, and build a bit map to tell you where there are, as offsets from the start of the image. Discard the image with the 256 byte offset, and put the one with the 0 byte offset on the SD card, along with the bitmap. When you load the image, load it starting at a 256 byte "page", and use the bitmap to adjust the value of each of those bytes to correct for the 256 byte "page corresponding to the actual load address.
It's a PITA, but does work with many processor architectures. With many (perhaps most) modern CPUs, there will very likely be aspects of the target CPU instruction encoding that make it more complicated, possibly by a lot.

Regards,
Ray L.

If Fuzzy was capable of that in the first place, it wouldn't have been necessary to ask this question.

You claim that modifying the linker script to manage the build process to exclude a new section of RAM is easier than...managing the build process with new sections of RAM? What am I missing?

My process would be to find the (existing) linker script that defines memory, which should contain a line that looks like:

   dram0_0_seg :                         org = 0x3FFE8000, len = 0x14000

I would change that to:

    dram0_0_seg :                         org = 0x3FFE8000, len = 0x13000

And I'd have a chunk of RAM at (0x3FFFB000) where I could put absolute-positioned variables (all painstakingly defined in a .h file somewhere.)

That's it. Regular sketches would still compile (with 4k less RAM available), and I wouldn't have to delve into the Arduino build process or commands at all.

Defining a new "shared, absolute-located memory" section would remove the "painstaking" part of variable definition, but you'd need to modify the linker commands used by Arduino IDE, AND the linker scripts, and figure out how to avoid having them garbage-collected (not SURE that's an issue, but it could be.)

Jiggy-Ninja:
I'm confused. You claim that modifying the linker script to manage the build process to exclude a new section of RAM is easier than...managing the build process with new sections of RAM? What am I missing?

Read it again. What I suggested does not require touching the build process, other than to force a second build with the base address shifted by 256 bytes. That is about as trivial a change as you can make - a single character edit to the memory segment definition. It does require post-processing the binary to create a relocation map that allows the code to be re-located on-the-fly as it's being read off the SD card. Whether that is practical with the process on the ESP, I have no idea. But this was a common, and relatively simple, way to handle run-time code relocation in the old days.

Regards,
Ray L.

westfw:
My process would be to find the (existing) linker script that defines memory, which should contain a line that looks like:

** lots of other stuff snipped for brevity **

I don't think that will be adequate for the parent program, and I certainly don't think it will work for the program being loaded off the SD card.

The parent program requires the following 3 things to be reserved and placed at known locations in the RAM:

  1. Buffer for the child program's RAM.
  2. Buffer for the child program's ROM.
  3. Struct containing pointers to the resources being made available to the child.

That's only 3 sections of RAM that need to be added to the linker script.

RayLivingston:
Read it again. What I suggested does not require touching the build process, other than to force a second build with the base address shifted by 256 bytes. That is about as trivial a change as you can make - a single character edit to the memory segment definition. It does require post-processing the binary to create a relocation map that allows the code to be re-located on-the-fly as it's being read off the SD card. Whether that is practical with the process on the ESP, I have no idea. But this was a common, and relatively simple, way to handle run-time code relocation in the old days.

Regards,
Ray L.

That relocation map would only be valid for that specific binary though, right?

And if you're capable of compiler for different memory start addresses, you might as well just compile for the known start locations of the parent program and not a random offset.

The more I think about this though, the more I wonder if it's really necessary. You might be better off writing a bootloader that flashes the program off of the SD card.

I don't think that will be adequate for the parent program,

No, of course not. It ONLY addresses the original question of shared data at absolute memory addresses.

In order to actually compile and run binaries from a flash chip, you need to do the following:

  1. arrange for the code to be position-independent, or arrange for it to be relocated, or arrange for it to be run at a different fixed address (in RAM) than normal binaries (The tensilica core has various PC-relative address modes, so this might be possible. I couldn't quickly tell whether -fPIC gcc option works.)
  2. similar for private data from the loaded binary; I guess if the binary is designed to run from RAM, you could reserve a section of RAM (in the parent) that was supposed to contain all the code AND data for the "child" program, and they'd all fit in a fixed address space. (But this would require a different setup than normal ESP8266 programs.)
  3. Some sort of scheme to prevent the child program from duplicating the startup code of the parent, or interfering with the peripherals, vectors, and other "global resources" that the parent is dealing with. This is essentially a new "board type" (if you were planning on using Arduino), and perhaps a new gcc target. (like the differences between "arm-eabi-none-gcc" and "arm-eabi-gcc") The child can't interfere with the Espressif ROM code, either.

And you'd probably also want:
4) Some sort of mechanism for the child program to make "system calls" to the parent in order to get access to particular services.

This is all "known technology." You're essentially looking at duplicating early microprocessor "operating system" like CP/M and MSDOS 3 (or earlier.) Glorified "program loaders" that provided common services. (and they ran in less than 64k, so the 64k of instruction RAM + lots of ROM/Flash in an ESP8266 should be "enough".) But it is also, to a large extent, "forgotten technology" - most programming these days makes used of a full-function operating system with memory mapping/protection (ie Windows or linux), or is designed for deep embedding in ROM/Flash (FreeRTOS/etc.) Not for the faint of heart. Finding or writing an interpreter that will work for you is probably a better solution.

That's why I think a bootloader that reflashes from the SD card is probably the easiest choice.

el_supremo:
Nope, that's not going to work. You can't call an array as a function.

An example of that which can't be done:

(Sort of) Running an Arduino Program Stored in Memory