Microcontroller with one terrabyte program

Other Hardware 3rd Party Boards
1

I've ordered an ESP8266 wifi board off eBay along with an SD card reader that connects over SPI.

I can put a one terrabyte SD card inside the SD card reader.

I'm thinking of writing a very small program that loads the machine code of functions from the SDcard into the microcontroller's program memory, swapping them in and out as they're needed. So basically I could have a very big program on the SDcard (as big as a terrabyte!).
Or maybe FAT32 would limit me to 4 gigs but that's still huge

Of course the machine code I load from the SD card would have to be position-independent. Also if the function calls another function, then I'll have to load the two of them in together, or alternatively I could have a global map of function names to function addresses, something like:

map<  string , void(*)(void)  > funcs;

void Func1(void)
{
    void (*const Func2)(void) = funcs["Func2"];

    Func2();
}

So I could swap functions in and out as I need them.

I bet I can get this working. Has anyone done anything like this before?

2

Why do you need to run such a huge program on a small mcu? :slight_smile: The art of programming is to make a short code for a long tasks :slight_smile:

3

That makes your code horribly slow. Each byte takes at least 16 clock ticks to transfer. And how do you want to swap the data in and out?

Do you also want to write a compiler for such code?

For good reason the PC architecture offers much hardware support for such a system. Just the first 8086 had hardware for support of virtual/segmented memory and overlays. You may get much inspiration from the 8086 manuals and MSDOS programmers handbooks.

Of course you'll learn a lot during the implementation of your intended system. But IMO it comes too late for nowadays big CPUs and too early or dead end for MCUs.

4

This is an intriguing idea, and in the realm of computing, similar concepts exist. For instance, demand paging is a concept in OS where only portions of a program are loaded into memory as required, while the rest stays on disk, and they are swapped in and out as needed. However, for microcontrollers, this isn't a conventional approach for a few reasons:

  1. Memory Management: Microcontrollers, especially simpler ones like the ESP8266, don't have MMUs (Memory Management Units). Implementing such dynamic code loading and execution would be a significant challenge.

  2. Performance Overhead: Reading from an SD card is orders of magnitude slower than accessing SRAM or flash memory. Any real-time or timing-sensitive operations would be highly impacted.

  3. Security: Loading and executing code from an SD card poses a security risk. An attacker with access to the SD card could potentially replace or alter the machine code of functions.

  4. Complexity: Managing the swapping of machine code in and out of memory while ensuring the proper function is always available would add significant complexity to your firmware.

  5. Position Independent Code (PIC): While creating position-independent code is possible for some platforms, doing so might be more complex for microcontroller targets. This is especially the case if you want to ensure every piece of your code, including library functions, is position independent.

  6. Limited Flash Writes: Flash memory on microcontrollers has a limited number of write cycles. Continuously writing and erasing will wear it out sooner.

That said, while the idea is ambitious and unconventional for microcontrollers, it's not entirely impossible. However, it's worth noting that it would likely be more of a proof-of-concept or an academic exercise than a practical solution for most applications. If you're looking for a fun challenge, and you're aware of the limitations and complexities, then go for it!

Regarding your question about whether anyone has done this before: while similar mechanisms exist in the world of general-purpose computers, I'm not aware of such an implementation specifically for microcontrollers. It's an uncharted territory that you might be pioneering!

5

I've had another idea about how to go about this.

Back when I was in college studying computer science, for my final year project I wrote a simulator for the most simple 8-bit microcontroller that I could find. So basically I wrote a desktop PC program to simulated a CPU with registers and volatile memory. I also wrote it to read binary firmware files and parse the machine code into instructions, and I executed the instructions on my simulated CPU.

So here's what I might do: Of all the GNU g++ cross-compilers that are available, I will pick the one that has the most simple CPU architecture and instruction set (maybe something like the Motorola M6800), and I will write my test program in C++ and use g++-13-m68k-linux-gnu to build it.

I will also write a program for the Arduino that simulates the CPU in question. My program will read machine code on demand from the SD card and run it in my simulated CPU. Whenever it tries to jump to an address that isn't currently loaded, I'll fetch the machine code from the SD card.

Wearing out the SD card isn't a concern as I'll only be reading from it.

This will be fun.

6

What sort of problems will you be very slowly solving with that enormous code base?

7

Sure, they did it for years... back in the sixties when RAM was extremely expensive.
Now, in 2023, why are you doing this? Note that "just because I can" is a perfectly valid answer, but I can't think of any other reason.

8

I had done this (referred to as code overlays) back in the 1970's and the 1980's.

In the 1970's case, it was handled by the linker. I had to first define overlay regions and the groups of functions that swap in each region. The linker then calculated the region sizes and added code so that a call to a function in a region would actually go to a helper function that would check to see that the right overlay group was loaded and then jump into that region. Functions in an overlay region could not call other functions in the region unless they were in the same group of files. It was a nice feature of RT11 on a Digital Equipment Corporation PDP-11.

In the 1980's I was doing similar things with a single swap region writing Programs in Forth that would not fit in the computer memory.

Thankfully, virtual memory came about and made all this chicanery a thing of the past. Personally, I can't imagine a 1TB program and if I had it I certainly wouldn't be running it on a microcontroller.

1 Like
9

This has been floating around for awhile, but apparently someone successfully booted Linux on an Atmega1284 (8-bit microcontroller) that was running ARM emulation code and using an external RAM chip with SD as a "hard drive".

https://hackaday.com/2012/03/28/building-the-worst-linux-pc-ever/

More recently there's an ESP32 running a RISC-V emulator running Linux.

https://hackaday.com/2021/07/21/its-linux-but-on-an-esp32/

The latter is probably a significant performance upgrade compared to the x386 SCO Unix machine that was my first *nix experience circa 1990.

10

Just now I sent an email to Dmitry, the guy who got Linux working on an 8-Bit microcontroller:

Dmitry

You wrote on your electronics website that your 8-bit microcontroller takes hours to boot Linux, but that it's surprisingly usable once it's booted.

I see that you've put some invalid opcodes in your CPU emulator in order to give Linux a way to communicate directly with your emulator.

I have a suggestion:
Add a new invalid opcode that freezes the emulator and takes a snapshot of the CPU registers, and also of the entire contents of the RAM stick. Save this snapshot to the SD card. When the microcontroller powers on, have it check the SDcard to see if there's a saved snapshot, and load the snapshot into the RAM stack and the emulator, then just set the emulator running. This should allow you to have Linux booted in just a minute or two.

The most impressive feat I've heard of in my life is a man named Fauja Singh who at the age of 101 ran the London marathon and finished it in 7hr 49min. I contacted his temple in England and asked them if I could fly over there some Sunday from Ireland just to shake his hand and then I would get on a flight back home. When I discovered yesterday that you got Linux booted up and usable on an 8-Bit microcontroller, I had to think whether Fauja Singh was still in my #1 slot. Today I still haven't made my mind up.

My name's Thomas and I'm from Ireland. I work a 9 to 5 programming the microcontrollers inside microscopes, and I post under the pseudonym 'Frederick Virchanza Gotham' to programming mailing lists and forums (sometimes as 'Frederick Gotham' or 'Virchanza').

Hope all is well,

Thomas

11

There was Andrew Tanenbaum's Minix as a precedent of Linux even on 8 bit microprocessors.

Emulation is a requirement in a Harvard architecture. But I doubt that emulation of a 32 bit controller on a 8 bit controller is a good idea. In detail with the ARM/THUMB fork implementation. IMO FORTH were a more efficient model with as many primitives as possible built into the emulator. With also compiler functions resident the system could run source code immediately (kind of JIT compilation).

12

If you go the simulator route, also look at this: C-like interpreter that actually fits and runs inside most Arduino chips: wrench

You'll probably have recognised that the contribution from @afhomon (post #4 here) was pasted from ChatGPT.

13

I suppose the same

14

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.