Go Down

Topic: SD Card bootloader (Read 705 times) previous topic - next topic

pert

I looked at the Optiboot section of the one of the core Github repos where they go through whats needed to build the bootloader.  I didn't understand any of it.
If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.

ShermanP

If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.
Thanks but I don't think that would be productive.  It would be way over my head.  The Naken assembler I used for the TI stuff, which also does AVR8 by the way, just assembles an input code file and the include files, and spits out the hex file to flash to the device.  None of this linker rubbish.  :-)  And the TIs don't have fuses.


pert

All you need to do to compile optiboot (or any other AVR bootloader I've ever encountered) is make sure avr-gcc (which is included with the Arduino IDE) is in the path, and then run the command:
Code: [Select]
make {target name}
from the folder containing the optiboot source code, where {target name} is the target you are compiling for. For example, if you wanted to compile optiboot for ATmega328P running at 16 MHz, you run this command:
Code: [Select]
make atmega328

The makefile automatically handles the entire compilation process, so there's no need to get all flustered about the linker.

I very much doubt that's over your head.

ShermanP

By the way, the 137 bytes appear to be just filling in the reset and interrupt vectors, and pointing all of them to little routines that shut everything down via endless loops.  Presumably GCC by itself would not add that.

The other thing I need to research is fuses.

I spent some time looking at AVR assembly, and of course it's completely different from TI.  8-bit vs 16, Harvard vs von Neumann, plus the TIs are kinda a little bit CISC.  So translating the TI code may not be a good idea.  But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI.  Then we could see what that looks like size-wise.

Juraj

#19
Apr 19, 2019, 05:12 pm Last Edit: Apr 19, 2019, 05:13 pm by Juraj
But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI.  Then we could see what that looks like size-wise.

that would be the mentioned avr_boot SD bootloader

https://github.com/zevero/avr_boot/blob/master/main.c
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

ron_sutherland

#20
Apr 19, 2019, 07:32 pm Last Edit: Apr 19, 2019, 07:39 pm by ron_sutherland
Quote
8-bit vs 16, Harvard vs von Neumann,
Harvard with an 8-bit data and a separate 16-bit instruction path (not shared), vs. von Neumann with a 16-bit path that is shared for both data and instructions. Harvard can use the data path to connect registers and/or ALU while at the same time using the instruction path (e.g., during a single clock cycle). von Neumann has to fetch the instruction and then during a succeeding clock use the same path to connect registers and/or ALU.

Quote
137 bytes appear to be just filling in the reset and interrupt vectors
That is an interrupt table populated by avr-gcc.

https://www.nongnu.org/avr-libc/user-manual/group__avr__interrupts.html

may want to look at this old topic though it goes off the rails.

https://www.avrfreaks.net/forum/avr-gcc-how-remove-interrupt-table

The linker options optiboot use do that; they remove the ISR table, and they are better explained here.

https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html
I use an R-Pi Zero on an RPUpi shield to have a tool-chain at the network edge.

ShermanP

I understand about the speed advantage of Harvard, but in this case that doesn't matter because the speed of SPI to the SD card is going to determine how long the update takes.  And it does appear that vonNeumann takes up less coding space since you don't have to run everything through the registers.  In any case, the real problem for me is that it's just completely different, and I really don't want to have to learn AVR assembly.

I spent an hour with the 328P datasheet this morning, and I think this is not going to work for me.  We already have an SD card bootloader, and the only thing wrong with it is it's too big.  But there's no evidence that having a smaller one would really benefit anyone.

So I think I'm done with this.

ron_sutherland

It is all good; I use to like assembly but that was a long, long time ago, I do not even know one AVR instruction. I am going to add a comment or question since I have been digging into this rabbit hole.

While looking at the bootloader linker options that others have used it seems that optiboot does not have a setting to place a stack. Does optiboot not use a stack? How is that even possible? Is it that the AVR has so many registers connected to the data bus that it can run stackless for this task?
I use an R-Pi Zero on an RPUpi shield to have a tool-chain at the network edge.

pert

there's no evidence that having a smaller one would really benefit anyone.
I very much disagree. If we could get it down to 2 kB, we could add support for more microcontrollers. It would also free up memory for the user application on the already supported parts. 2 kB extra flash on a chip with only 32 kB total is a big deal! Even if we can't get it small enough to fit in the next smaller boot section, getting the code smaller leaves room to add more features to the bootloader. In the case of avr_boot, that could be serial support.

Of course I'm not saying I expect you to do that, but only that I do think it would be worthwhile for someone to do it.

So I think I'm done with this.
Understandable. Well, I hope we were able to have a nice discussion here at least.

ShermanP

I very much disagree. If we could get it down to 2 kB, we could add support for more microcontrollers. It would also free up memory for the user application on the already supported parts. 2 kB extra flash on a chip with only 32 kB total is a big deal! Even if we can't get it small enough to fit in the next smaller boot section, getting the code smaller leaves room to add more features to the bootloader. In the case of avr_boot, that could be serial support.

Of course I'm not saying I expect you to do that, but only that I do think it would be worthwhile for someone to do it.

I keep coming back to this, but then end up with nowhere to go.  I spent a good bit of time looking at avr_boot, but since I never became proficient in C, much of it makes no sense to me.  But I wonder why the code is so much bigger than by MSP430 version.  It could be because it's AVR8, and that just takes more code, or because it's mostly C instead of assembler, although I'm told C is pretty efficient, or because of the need to support so many processors and Arduinos, although it seems that would all be taken care of in the build process, or, my favorite, because avr_boot still makes considerable use of libraries, like for FAT, and inevitably the functions that end up being included do more than you really need, or do it less efficiently.

If getting the size under 2K is important, to me that just cries out for doing it in assembler, using no libraries, but I take your point that it then becomes an opaque  blob of code for most people, so not much use to anyone else.   However, I don't know how many people would be successful wading their way through Optiboot, even in C.

Attached is a schematic showing modifications to the Arduino, and to the widely available level-shifting microSD module, which would let the bootloader detect the presence of an SD card without sending commands and waiting for a response, and without using another pin.  It basically works like my TI version, with a little jumping through hoops to avoid sourcing 5V to the SD card.  I guess you wouldn't approve of requiring such modifications, and I would probably agree.

Anyway, I guess the problem for me is that I don't really know C++ or AVR8 assembler, so I don't know what I would do to go forward with this.  All I know is the steps needed to deal with SD and SDHC, FAT16 and FAT32, which I think is what should be supported, not MMC or FAT12 anymore, or SDXC.  I should also say that my version of this has no capabilty to navigate cluster chains, which makes the code a lot shorter, but does introduce certain risks, particularly concerning the root directory in FAT32.  And my TI version also requires the HEX file to be pre-converted to binary so the bootloader doesn't have to parse a hex file.  So it may be that my version is smaller because it does a lot less.


pert

The other three SD bootloaders listed in the avr_boot readme claim to fit in a 2 kB boot section:
https://github.com/zevero/avr_boot#alternatives
I don't know what it is that makes avr_boot larger than those. Maybe it's suboptimal code. Maybe it has functionality that the others don't. At least the alternatives do prove that it's possible to fit an AVR SD bootloader in 2 kB.

I got involved with avr_boot because there was a request to add Boards Manager installation support in the Arduino IDE. At that time, Boards Manager was fairly new and I was sort of specializing in helping people add this capability to their projects. I liked the maintainer's willingness to make the project accessible to the average Arduino user. The alternatives send you off with a makefile and a "good luck". At the time, avr_boot was being actively maintained by zevero, while the alternatives hadn't had any activity in years.

And my TI version also requires the HEX file to be pre-converted to binary so the bootloader doesn't have to parse a hex file.  So it may be that my version is smaller because it does a lot less.
Same requirement with avr_boot, https://spaces.atmel.com/gf/project/sdbootloader/, and I believe http://www.mikrocontroller.net/articles/MMC/SD_Bootloader_f%C3%BCr_AT_Mega too.

Interestingly, 2boots claims to work from HEX files, has both SD and serial capability, and fits in 2 kB!

Juraj

#26
Apr 23, 2019, 11:10 am Last Edit: Apr 23, 2019, 11:22 am by Juraj
Interestingly, 2boots claims to work from HEX files, has both SD and serial capability, and fits in 2 kB!
but doesn't work with available (micro) SD cards. only with some old large cards
"Supported are FAT16 formatted cards up to 2GB in size (no SDHC yet)"

avr_boot expects binary file on SD

the other mentioned SD bootloaders are not finished

I tested the SD bootloaders for use with ArduinoOTA library.
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

ShermanP

but doesn't work with available (micro) SD cards. only with some old large cards
"Supported are FAT16 formatted cards up to 2GB in size (no SDHC yet)"

avr_boot expects binary file on SD

the other mentioned SD bootloaders are not finished

I tested the SD bootloaders for use with ArduinoOTA library.
Perhaps it would be possible to fix 2boots so it will work with SDHC and FAT32.  The change for SDHC is that the starting address to read from is the sector number, as opposed to the byte number in SD.  So really it comes down to doing it all in sectors, but then multiplying by 512 if it's SD.  And even the FAT32 vs FAT16 differences aren't all that much in coding terms.  And we could delete anything that's only there for MMC or FAT12.

The big issue is whether or not to implement cluster chaining.  I managed to talk myself into not including it in my bootloader since the smallest likely cluster size was 16K, which would be big enough to hold the update file (in binary).  So in theory all files would be stored in the consecutive sectors of one cluster, so never a need to go to the next cluster.  The problem is the root directory in FAT32, which is just another cluster chain.  But even there, with a cluster size of 32K, that's 1024 directory entries, and I figured you could give instructions not to use the card for any other purpose, so it would likely never have more than a few dozen entries.

I'll take a look at 2boots and see if I can figure out what he did.

Juraj

Perhaps it would be possible to fix 2boots so it will work with SDHC and FAT32.  The change for SDHC is that the starting address to read from is the sector number, as opposed to the byte number in SD.  So really it comes down to doing it all in sectors, but then multiplying by 512 if it's SD.  And even the FAT32 vs FAT16 differences aren't all that much in coding terms.  And we could delete anything that's only there for MMC or FAT12.

The big issue is whether or not to implement cluster chaining.  I managed to talk myself into not including it in my bootloader since the smallest likely cluster size was 16K, which would be big enough to hold the update file (in binary).  So in theory all files would be stored in the consecutive sectors of one cluster, so never a need to go to the next cluster.  The problem is the root directory in FAT32, which is just another cluster chain.  But even there, with a cluster size of 32K, that's 1024 directory entries, and I figured you could give instructions not to use the card for any other purpose, so it would likely never have more than a few dozen entries.

I'll take a look at 2boots and see if I can figure out what he did.

16 kB? a sketch with Ethernet and SD library is larger. my project has a 71 kB binary build for a Mega
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

ron_sutherland

After thinking on this, the direction I am going to try is to use avrdude from an R-Pi Zero to do both serial booload and ICSP from a relatively small handheld but power hungry setup. It will have spring probes on both ends. I modified a little level shift board I was using with my Uno that does the ISP sketch.

https://github.com/epccs/Driver/tree/master/ICSP

The exact details for running the uploader are yet to be worked out, but for me, it will probably involve a remote login and then running a Python program with a loop that waits for a switch and then operates a makefile rule based on the working project directory then does that all again. It can build from source (I use GitHub), so on the ICSP tool's host (samba share or whatever) I can even edit it and don't need to keep track of what SD card goes with what project.

Power is a problem, I will need to carry a car battery around to run such a thing in the back yard.
I use an R-Pi Zero on an RPUpi shield to have a tool-chain at the network edge.

Go Up