SD Card bootloader

ShermanP:
Ok, so what's special about bootloaders is where they are stored. I understand the problem. One option I've used with the TI parts when I need to tamper with locked sections of memory is to write an installer that's loaded into main memory in the normal way. It contains an image of what I need to write to locked memory, and my installer unlocks the locked section, erases it, copies the image to it, locks it again, sets the reset vector to whatever is needed, then shuts down. So flashing the installer does nothing. You have to power up again so it will run and do its thing. Is that possible in the AVR world? Wasn't there a recent change to Optiboot that implements changing flash from flash? Well, maybe not changing the bootloader from flash.

not all AVR MCUs have the same architecture. in ATmega MCU, if it supports bootloader, only the code in bootloader section can write to flash. It is in Read-While-Write part of the flash so in theory a function running in last page could overwrite the bootloader.

The Arduino IDE also supports assembly if you prefer. You just need to put it in a file of the sketch with the .S extension.

If I delete the contents of the core library folder, the minimal sketch memory usage goes down to 0 bytes. It's quite possible to create a custom hardware package that will have no overhead for the "Arduino magic", but I don't know how you can avoid those 138 bytes of overhead when using the standard Arduino AVR Boards hardware package.

I looked at the Optiboot section of the one of the core Github repos where they go through whats needed to build the bootloader. I didn't understand any of it.

I'm not sure there's anything to be done here. I have the steps needed to initialize the SD card and negotiate the FAT file system to load in the file contents, but translating that into a bootloader is another matter.

The magic that controls the toolchain to build a bootloader was most accessible for me to understand by learning some Makefile stuff. Makefiles are just rules that command the toolchain, so they are not magic at all.

I do not find that Makefile to be accessible, so I learned a few things and then derived one that was more accessible to me.

The things that allow it to work are the linker and compiler options and flags, and where to place the bootloader in memory. Having that as IDE clutter is probably not a good idea, the target audience is wrong (look at the unusable Atmel Studio trainwreck for a reference of what it might look like). I use an Uno loaded with the ISP sketch and run two commands "make fuse" and then "make isp". I also use the EEPROM so have given myself an option to recover it.

ShermanP:
I looked at the Optiboot section of the one of the core Github repos where they go through whats needed to build the bootloader. I didn't understand any of it.

If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.

pert:
If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.

Thanks but I don't think that would be productive. It would be way over my head. The Naken assembler I used for the TI stuff, which also does AVR8 by the way, just assembles an input code file and the include files, and spits out the hex file to flash to the device. None of this linker rubbish. :slight_smile: And the TIs don't have fuses.

All you need to do to compile optiboot (or any other AVR bootloader I've ever encountered) is make sure avr-gcc (which is included with the Arduino IDE) is in the path, and then run the command:

make {target name}

from the folder containing the optiboot source code, where {target name} is the target you are compiling for. For example, if you wanted to compile optiboot for ATmega328P running at 16 MHz, you run this command:

make atmega328

The makefile automatically handles the entire compilation process, so there's no need to get all flustered about the linker.

I very much doubt that's over your head.

By the way, the 137 bytes appear to be just filling in the reset and interrupt vectors, and pointing all of them to little routines that shut everything down via endless loops. Presumably GCC by itself would not add that.

The other thing I need to research is fuses.

I spent some time looking at AVR assembly, and of course it's completely different from TI. 8-bit vs 16, Harvard vs von Neumann, plus the TIs are kinda a little bit CISC. So translating the TI code may not be a good idea. But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI. Then we could see what that looks like size-wise.

ShermanP:
But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI. Then we could see what that looks like size-wise.

that would be the mentioned avr_boot SD bootloader

8-bit vs 16, Harvard vs von Neumann,

Harvard with an 8-bit data and a separate 16-bit instruction path (not shared), vs. von Neumann with a 16-bit path that is shared for both data and instructions. Harvard can use the data path to connect registers and/or ALU while at the same time using the instruction path (e.g., during a single clock cycle). von Neumann has to fetch the instruction and then during a succeeding clock use the same path to connect registers and/or ALU.

137 bytes appear to be just filling in the reset and interrupt vectors

That is an interrupt table populated by avr-gcc.

https://www.nongnu.org/avr-libc/user-manual/group__avr__interrupts.html

may want to look at this old topic though it goes off the rails.

https://www.avrfreaks.net/forum/avr-gcc-how-remove-interrupt-table

The linker options optiboot use do that; they remove the ISR table, and they are better explained here.

I understand about the speed advantage of Harvard, but in this case that doesn't matter because the speed of SPI to the SD card is going to determine how long the update takes. And it does appear that vonNeumann takes up less coding space since you don't have to run everything through the registers. In any case, the real problem for me is that it's just completely different, and I really don't want to have to learn AVR assembly.

I spent an hour with the 328P datasheet this morning, and I think this is not going to work for me. We already have an SD card bootloader, and the only thing wrong with it is it's too big. But there's no evidence that having a smaller one would really benefit anyone.

So I think I'm done with this.

It is all good; I use to like assembly but that was a long, long time ago, I do not even know one AVR instruction. I am going to add a comment or question since I have been digging into this rabbit hole.

While looking at the bootloader linker options that others have used it seems that optiboot does not have a setting to place a stack. Does optiboot not use a stack? How is that even possible? Is it that the AVR has so many registers connected to the data bus that it can run stackless for this task?

ShermanP:
there's no evidence that having a smaller one would really benefit anyone.

I very much disagree. If we could get it down to 2 kB, we could add support for more microcontrollers. It would also free up memory for the user application on the already supported parts. 2 kB extra flash on a chip with only 32 kB total is a big deal! Even if we can't get it small enough to fit in the next smaller boot section, getting the code smaller leaves room to add more features to the bootloader. In the case of avr_boot, that could be serial support.

Of course I'm not saying I expect you to do that, but only that I do think it would be worthwhile for someone to do it.

ShermanP:
So I think I'm done with this.

Understandable. Well, I hope we were able to have a nice discussion here at least.

pert:
I very much disagree. If we could get it down to 2 kB, we could add support for more microcontrollers. It would also free up memory for the user application on the already supported parts. 2 kB extra flash on a chip with only 32 kB total is a big deal! Even if we can't get it small enough to fit in the next smaller boot section, getting the code smaller leaves room to add more features to the bootloader. In the case of avr_boot, that could be serial support.

Of course I'm not saying I expect you to do that, but only that I do think it would be worthwhile for someone to do it.

I keep coming back to this, but then end up with nowhere to go. I spent a good bit of time looking at avr_boot, but since I never became proficient in C, much of it makes no sense to me. But I wonder why the code is so much bigger than by MSP430 version. It could be because it's AVR8, and that just takes more code, or because it's mostly C instead of assembler, although I'm told C is pretty efficient, or because of the need to support so many processors and Arduinos, although it seems that would all be taken care of in the build process, or, my favorite, because avr_boot still makes considerable use of libraries, like for FAT, and inevitably the functions that end up being included do more than you really need, or do it less efficiently.

If getting the size under 2K is important, to me that just cries out for doing it in assembler, using no libraries, but I take your point that it then becomes an opaque blob of code for most people, so not much use to anyone else. However, I don't know how many people would be successful wading their way through Optiboot, even in C.

Attached is a schematic showing modifications to the Arduino, and to the widely available level-shifting microSD module, which would let the bootloader detect the presence of an SD card without sending commands and waiting for a response, and without using another pin. It basically works like my TI version, with a little jumping through hoops to avoid sourcing 5V to the SD card. I guess you wouldn't approve of requiring such modifications, and I would probably agree.

Anyway, I guess the problem for me is that I don't really know C++ or AVR8 assembler, so I don't know what I would do to go forward with this. All I know is the steps needed to deal with SD and SDHC, FAT16 and FAT32, which I think is what should be supported, not MMC or FAT12 anymore, or SDXC. I should also say that my version of this has no capabilty to navigate cluster chains, which makes the code a lot shorter, but does introduce certain risks, particularly concerning the root directory in FAT32. And my TI version also requires the HEX file to be pre-converted to binary so the bootloader doesn't have to parse a hex file. So it may be that my version is smaller because it does a lot less.

The other three SD bootloaders listed in the avr_boot readme claim to fit in a 2 kB boot section:

I don't know what it is that makes avr_boot larger than those. Maybe it's suboptimal code. Maybe it has functionality that the others don't. At least the alternatives do prove that it's possible to fit an AVR SD bootloader in 2 kB.

I got involved with avr_boot because there was a request to add Boards Manager installation support in the Arduino IDE. At that time, Boards Manager was fairly new and I was sort of specializing in helping people add this capability to their projects. I liked the maintainer's willingness to make the project accessible to the average Arduino user. The alternatives send you off with a makefile and a "good luck". At the time, avr_boot was being actively maintained by zevero, while the alternatives hadn't had any activity in years.

ShermanP:
And my TI version also requires the HEX file to be pre-converted to binary so the bootloader doesn't have to parse a hex file. So it may be that my version is smaller because it does a lot less.

Same requirement with avr_boot, https://spaces.atmel.com/gf/project/sdbootloader/, and I believe http://www.mikrocontroller.net/articles/MMC/SD_Bootloader_für_AT_Mega too.

Interestingly, 2boots claims to work from HEX files, has both SD and serial capability, and fits in 2 kB!

pert:
Interestingly, 2boots claims to work from HEX files, has both SD and serial capability, and fits in 2 kB!

but doesn't work with available (micro) SD cards. only with some old large cards
"Supported are FAT16 formatted cards up to 2GB in size (no SDHC yet)"

avr_boot expects binary file on SD

the other mentioned SD bootloaders are not finished

I tested the SD bootloaders for use with ArduinoOTA library.

Juraj:
but doesn't work with available (micro) SD cards. only with some old large cards
"Supported are FAT16 formatted cards up to 2GB in size (no SDHC yet)"

avr_boot expects binary file on SD

the other mentioned SD bootloaders are not finished

I tested the SD bootloaders for use with ArduinoOTA library.

Perhaps it would be possible to fix 2boots so it will work with SDHC and FAT32. The change for SDHC is that the starting address to read from is the sector number, as opposed to the byte number in SD. So really it comes down to doing it all in sectors, but then multiplying by 512 if it's SD. And even the FAT32 vs FAT16 differences aren't all that much in coding terms. And we could delete anything that's only there for MMC or FAT12.

The big issue is whether or not to implement cluster chaining. I managed to talk myself into not including it in my bootloader since the smallest likely cluster size was 16K, which would be big enough to hold the update file (in binary). So in theory all files would be stored in the consecutive sectors of one cluster, so never a need to go to the next cluster. The problem is the root directory in FAT32, which is just another cluster chain. But even there, with a cluster size of 32K, that's 1024 directory entries, and I figured you could give instructions not to use the card for any other purpose, so it would likely never have more than a few dozen entries.

I'll take a look at 2boots and see if I can figure out what he did.

ShermanP:
Perhaps it would be possible to fix 2boots so it will work with SDHC and FAT32. The change for SDHC is that the starting address to read from is the sector number, as opposed to the byte number in SD. So really it comes down to doing it all in sectors, but then multiplying by 512 if it's SD. And even the FAT32 vs FAT16 differences aren't all that much in coding terms. And we could delete anything that's only there for MMC or FAT12.

The big issue is whether or not to implement cluster chaining. I managed to talk myself into not including it in my bootloader since the smallest likely cluster size was 16K, which would be big enough to hold the update file (in binary). So in theory all files would be stored in the consecutive sectors of one cluster, so never a need to go to the next cluster. The problem is the root directory in FAT32, which is just another cluster chain. But even there, with a cluster size of 32K, that's 1024 directory entries, and I figured you could give instructions not to use the card for any other purpose, so it would likely never have more than a few dozen entries.

I'll take a look at 2boots and see if I can figure out what he did.

16 kB? a sketch with Ethernet and SD library is larger. my project has a 71 kB binary build for a Mega

After thinking on this, the direction I am going to try is to use avrdude from an R-Pi Zero to do both serial booload and ICSP from a relatively small handheld but power hungry setup. It will have spring probes on both ends. I modified a little level shift board I was using with my Uno that does the ISP sketch.

The exact details for running the uploader are yet to be worked out, but for me, it will probably involve a remote login and then running a Python program with a loop that waits for a switch and then operates a makefile rule based on the working project directory then does that all again. It can build from source (I use GitHub), so on the ICSP tool's host (samba share or whatever) I can even edit it and don't need to keep track of what SD card goes with what project.

Power is a problem, I will need to carry a car battery around to run such a thing in the back yard.