SD Card bootloader

Other Hardware 3rd Party Boards
1

A search shows this has been attempted a few times over the years, but I've written a small SD card bootloader for another brand of processor, and thought that work might be of benefit to someone in the Arduino world if they have any interest in this idea. My bootloader was written for MSP430 processors, in assembler, and occupies 1K of flash. It supports SD and SDHC, FAT16 and FAT32.

It just seemed to me that you could fit both the SD card bootloader and Optiboot into the 2K bootloader area. Of course the AVR assembly would be completely different, as would the SPI modules, and there is a Harvard vs. von Neumann difference, but my code is pretty well commented as to the logic of dealing with the SD card and navigating through the file system. So I thought I would post here in case anyone wanted to take a look.

The basic idea is to make possible firmware updates without requiring any particular software, drivers or adapters. Just download the new firmware binary file, copy it to the SD card, insert the card, power up the Arduino, and the bootloader detects the card and copies the file contents into the program area.

2

Thanks for sharing!

I've been involved with this SD bootloader project for AVR:

It's not at all lightweight and uploading via serial is not officially supported (though a PR was submitted to add this), so yours sounds quite superior. That said, what avr_boot does have going for it is that it's very accessible to the average Arduino user. Many will gladly trade a few kB of flash for ease of use.

I'd love to see a superior alternative to avr_boot that still has an Arduino-style user-friendly focus, especially since the owner of the avr_boot repository seems to have mostly abandoned that project.

3

I would like to see an ICSP (e.g., OptiLoader) project that worked from an SD card.

4

ron_sutherland:
I would like to see an ICSP (e.g., OptiLoader) project that worked from an SD card.

GitHub - adafruit/Standalone-Arduino-AVR-ISP-programmer: A standalone programmer for mass-programming AVR chips

Wish granted:

Crossroads also sells finished standalone SD programmers:
http://www.crossroadsfencing.com/BobuinoRev17/Programmer.html

5

pert:
Thanks for sharing!

I've been involved with this SD bootloader project for AVR:
GitHub - zevero/avr_boot: Arduino Bootloader to Flash from SD Card
It's not at all lightweight and uploading via serial is not officially supported (though a PR was submitted to add this), so yours sounds quite superior. That said, what avr_boot does have going for it is that it's very accessible to the average Arduino user. Many will gladly trade a few kB of flash for ease of use.

I'd love to see a superior alternative to avr_boot that still has an Arduino-style user-friendly focus, especially since the owner of the avr_boot repository seems to have mostly abandoned that project.

My SD bootloader for the TI parts is intended for products that will need the ability to update firmware, and which can include a microSD port in the design. Also, some of the less expensive chips have very little flash memory to work with, and even less ram, so size could matter a lot. But I don't know how often you would see an Arduino-based product being used by someone who doesn't also have the IDE installed, so maybe the typical user would just use serial. In any case, it appears your bootloader is there for those who need the SD functionality. I'm very new to Atmel and Arduino, and not competent to be writing bootloaders. I just thought that the magic number for bootloader size was 2K, and I was wrong about that, but wanted to make the point that it might be possible to do a version of SD in about 1K, which would leave room for Optiboot within the old 2K bootloader size.

My code is small mainly because I implement only what's absolutely needed for the SD card and for FAT. And I impose some restrictions to help with that. The one that still bothers me a bit is that a file has to be stored in consecutive sectors on the drive. I don't implement any cluster chaining since many of the TI processors don't have enough ram to bring in much of the FAT table. The most I bring in at a time from any sector is 64 bytes, and the actual firmware update is written one byte at a time as the bytes come in from the SD card. I don't know how much of this would be relevant with the Atmel chips.

It occurred to me last night that another difference between the two systems is that the TI chips are 16-bit vonNeumann, which may allow the code to be considerably smaller than AVR. Don't really know for sure.

With respect to the other comments, my TI bootloader and your avr_boot aren't the same thing as a separate SD-based programmer. An SD bootloader eliminates the need for such a device.

In case I want to pursue this further, can you tell me how you use the IDE to write a bootloader? Do you just not use the setup() and loop() routines so you get pretty much pure code without all the background Arduino stuff? I was thinking it might be possible to write my code in C++ for Energia, and just see how much size difference it makes, then use the same C++ code in the Arduino IDE, if that's possible, and see what that produces size-wise. I have a feeling both would be a good bit larger than 1K.

6

ShermanP:
In case I want to pursue this further, can you tell me how you use the IDE to write a bootloader? Do you just not use the setup() and loop() routines so you get pretty much pure code without all the background Arduino stuff? I was thinking it might be possible to write my code in C++ for Energia, and just see how much size difference it makes, then use the same C++ code in the Arduino IDE, if that's possible, and see what that produces size-wise. I have a feeling both would be a good bit larger than 1K.

creating a bootloader is not an Arduino topic. you can't use IDE for it. it should be written in C. see bootloader source codes on github.

7

Thanks, @pert, I have taken some notes and will give gammon's set up a try.

Maybe I'm wrong, but I don't think that it is always the case that von Neumann architecture is advantageous compared to Harvard with regard to code size. The AVR has a pretty much complete instruction set, so you don't need to use a bunch of them to do what is a common task. Thus the arguments of reduced code with CISC vs. RISC may not hold so well; I have no proof of this; it is just a hunch; hence the reason I may be wrong.

8

ShermanP:
I don't know how often you would see an Arduino-based product being used by someone who doesn't also have the IDE installed, so maybe the typical user would just use serial.

I'll admit that, despite having spent a lot of time contributing to the avr_boot project, I haven't actually found a need for it in one of my own projects. I could see uses for it though. For example, you create a handheld gaming console and load different games from SD cards like mini Gameboy cartridges. Perhaps, you could use it to load different visualizations on an addressable LED art piece in the field.

ShermanP:
I just thought that the magic number for bootloader size was 2K

It certainly is on some chips. If avr_boot fit in a 2 kB boot section, we could add support for more chips, such as the ATmega168.

Juraj:
you can't use IDE for it.

I'm not at all convinced of that. I think it would be way cool to be able to play with bootloader source code, compile, and flash it all from the Arduino IDE. It would require a customized hardware package to accomplish the flashing part of that. I did make a start at investigating this possibility a while back, but haven't spent enough time on the project to get very far with it yet.

9

pert:
I think it would be way cool to be able to play with bootloader source code, compile, and flash it all from the Arduino IDE. It would require a customized hardware package to accomplish the flashing part of that. I did make a start at investigating this possibility a while back, but haven't spent enough time on the project to get very far with it yet.

I don't understand why you would need a custom hardware package for flashing. You can flash a bootloader to an Atmel chip now using a regular programmer or another Arduino-as-ISP. Why couldn't either programming method be used to flash an SD card bootloader?

With respect to using the IDE to code a bootloader, I was under the impression that if you deleted all the setup() and loop() stuff, and went back to plain C or C++ code, with a main(), the IDE would not add all the usual Arduino management stuff, and would just let the GCC compiler do its regular thing. Of course that could be completely wrong. But I think even GCC would do things that you might not do if you were writing in assembler.

10

ShermanP:
I don't understand why you would need a custom hardware package for flashing. You can flash a bootloader to an Atmel chip now using a regular programmer or another Arduino-as-ISP. Why couldn't either programming method be used to flash an SD card bootloader?

When you compile a sketch in the Arduino IDE, the binary is saved to a temporary build folder with a randomized name. When you do a Tools > Burn Bootloader in the Arduino IDE, it flashes a binary file from a specific location specified in the board definition, which is certainly not going to be the temporary build folder. I don't see any way you could a non-customized hardware package to compile and burn a bootloader using the Arduino IDE exclusively, without needing to go dig around in your hard drive and copy files to the correct places every time.

The idea I have is to make it just as easy to compile and flash a bootloader as it is to upload a blink sketch. Of course the bootloader code is more complex than your average sketch, but this would let people skip over the whole toolchain, makefile part of the process (which may not be something they are interested in learning), and get straight to the programming part of playing with bootloaders.

ShermanP:
I was under the impression that if you deleted all the setup() and loop() stuff, and went back to plain C or C++ code, with a main(), the IDE would not add all the usual Arduino management stuff, and would just let the GCC compiler do its regular thing. Of course that could be completely wrong. But I think even GCC would do things that you might not do if you were writing in assembler.

It's certainly possible to do this with a custom hardware package. There's no reason using the Arduino IDE to compile some C or C++ code with GCC using needs to be any different from doing the same from a makefile. However, I don't know whether you can get rid of all the Arduino stuff with a standard package. Here is the most minimal Arduino sketch I know how to write:

// Prevent the Arduino IDE's automatic addition of an #include directive for Arduino.h
#if false
#include <Arduino.h>
#endif

// Override the definition of main() in the core library
int main(void) {}

When I compile that for Uno, I get:

Sketch uses 138 bytes (0%) of program storage space. Maximum is 32256 bytes.
Global variables use 0 bytes (0%) of dynamic memory, leaving 2048 bytes for local variables. Maximum is 2048 bytes.

I think that 138 bytes is from something in the core library, which is still getting compiled.

11

Ok, so what's special about bootloaders is where they are stored. I understand the problem. One option I've used with the TI parts when I need to tamper with locked sections of memory is to write an installer that's loaded into main memory in the normal way. It contains an image of what I need to write to locked memory, and my installer unlocks the locked section, erases it, copies the image to it, locks it again, sets the reset vector to whatever is needed, then shuts down. So flashing the installer does nothing. You have to power up again so it will run and do its thing. Is that possible in the AVR world? Wasn't there a recent change to Optiboot that implements changing flash from flash? Well, maybe not changing the bootloader from flash.

So I compiled your minimal code and disassembled the resulting .hex file. Seems to be a lot of jumping about for no purpose:

naken_util - by Michael Kohn
                Joe Davisson
    Web: http://www.mikekohn.net/
  Email: mike@mikekohn.net

Version: December 12, 2018

Loaded hexfile test.ino.hex from 0x0000 to 0x0089
Type help for a list of commands.

Addr    Opcode Instruction
------- ------ ----------------------------------
0x0000: 940c jmp 0x34
     0034
0x0002: 940c jmp 0x3e
     003e
0x0004: 940c jmp 0x3e
     003e
0x0006: 940c jmp 0x3e
     003e
0x0008: 940c jmp 0x3e
     003e
0x000a: 940c jmp 0x3e
     003e
0x000c: 940c jmp 0x3e
     003e
0x000e: 940c jmp 0x3e
     003e
0x0010: 940c jmp 0x3e
     003e
0x0012: 940c jmp 0x3e
     003e
0x0014: 940c jmp 0x3e
     003e
0x0016: 940c jmp 0x3e
     003e
0x0018: 940c jmp 0x3e
     003e
0x001a: 940c jmp 0x3e
     003e
0x001c: 940c jmp 0x3e
     003e
0x001e: 940c jmp 0x3e
     003e
0x0020: 940c jmp 0x3e
     003e
0x0022: 940c jmp 0x3e
     003e
0x0024: 940c jmp 0x3e
     003e
0x0026: 940c jmp 0x3e
     003e
0x0028: 940c jmp 0x3e
     003e
0x002a: 940c jmp 0x3e
     003e
0x002c: 940c jmp 0x3e
     003e
0x002e: 940c jmp 0x3e
     003e
0x0030: 940c jmp 0x3e
     003e
0x0032: 940c jmp 0x3e
     003e
0x0034: 2411 eor r1, r1
0x0035: be1f out 0x3f, r1
0x0036: efcf ldi r28, 0xff
0x0037: e0d8 ldi r29, 0x8
0x0038: bfde out 0x3e, r29
0x0039: bfcd out 0x3d, r28
0x003a: 940e call 0x40
     0040
0x003c: 940c jmp 0x43
     0043
0x003e: 940c jmp 0x0
     0000
0x0040: e080 ldi r24, 0x0
0x0041: e090 ldi r25, 0x0
0x0042: 9508 ret
0x0043: 94f8 cli
0x0044: cfff rjmp 0x44 (-1)

I think there's another reason why translating my code from assembler to C or C++ would increase the size. I routinely call subroutines at various points within the subroutine. So for example, most of the time I need to execute the subroutine with the same value going in, so I include loading that value in the subroutine itself. But one time I may need to run it with a different value, so I load it outside the subroutine, then call into the subroutine at a later point. Makes perfect sense in assembler, but I don't think C will let me do that.

12

ShermanP:
Ok, so what's special about bootloaders is where they are stored. I understand the problem. One option I've used with the TI parts when I need to tamper with locked sections of memory is to write an installer that's loaded into main memory in the normal way. It contains an image of what I need to write to locked memory, and my installer unlocks the locked section, erases it, copies the image to it, locks it again, sets the reset vector to whatever is needed, then shuts down. So flashing the installer does nothing. You have to power up again so it will run and do its thing. Is that possible in the AVR world? Wasn't there a recent change to Optiboot that implements changing flash from flash? Well, maybe not changing the bootloader from flash.

not all AVR MCUs have the same architecture. in ATmega MCU, if it supports bootloader, only the code in bootloader section can write to flash. It is in Read-While-Write part of the flash so in theory a function running in last page could overwrite the bootloader.

13

The Arduino IDE also supports assembly if you prefer. You just need to put it in a file of the sketch with the .S extension.

If I delete the contents of the core library folder, the minimal sketch memory usage goes down to 0 bytes. It's quite possible to create a custom hardware package that will have no overhead for the "Arduino magic", but I don't know how you can avoid those 138 bytes of overhead when using the standard Arduino AVR Boards hardware package.

14

I looked at the Optiboot section of the one of the core Github repos where they go through whats needed to build the bootloader. I didn't understand any of it.

I'm not sure there's anything to be done here. I have the steps needed to initialize the SD card and negotiate the FAT file system to load in the file contents, but translating that into a bootloader is another matter.

15

The magic that controls the toolchain to build a bootloader was most accessible for me to understand by learning some Makefile stuff. Makefiles are just rules that command the toolchain, so they are not magic at all.

I do not find that Makefile to be accessible, so I learned a few things and then derived one that was more accessible to me.

The things that allow it to work are the linker and compiler options and flags, and where to place the bootloader in memory. Having that as IDE clutter is probably not a good idea, the target audience is wrong (look at the unusable Atmel Studio trainwreck for a reference of what it might look like). I use an Uno loaded with the ISP sketch and run two commands "make fuse" and then "make isp". I also use the EEPROM so have given myself an option to recover it.

16

ShermanP:
I looked at the Optiboot section of the one of the core Github repos where they go through whats needed to build the bootloader. I didn't understand any of it.

If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.

17

pert:
If you're interested, I can provide specific instructions you can use to compile Optiboot. I just need to know which operating system you're using.

Thanks but I don't think that would be productive. It would be way over my head. The Naken assembler I used for the TI stuff, which also does AVR8 by the way, just assembles an input code file and the include files, and spits out the hex file to flash to the device. None of this linker rubbish. :slight_smile: And the TIs don't have fuses.

18

All you need to do to compile optiboot (or any other AVR bootloader I've ever encountered) is make sure avr-gcc (which is included with the Arduino IDE) is in the path, and then run the command:

make {target name}

from the folder containing the optiboot source code, where {target name} is the target you are compiling for. For example, if you wanted to compile optiboot for ATmega328P running at 16 MHz, you run this command:

make atmega328

The makefile automatically handles the entire compilation process, so there's no need to get all flustered about the linker.

I very much doubt that's over your head.

19

By the way, the 137 bytes appear to be just filling in the reset and interrupt vectors, and pointing all of them to little routines that shut everything down via endless loops. Presumably GCC by itself would not add that.

The other thing I need to research is fuses.

I spent some time looking at AVR assembly, and of course it's completely different from TI. 8-bit vs 16, Harvard vs von Neumann, plus the TIs are kinda a little bit CISC. So translating the TI code may not be a good idea. But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI. Then we could see what that looks like size-wise.

20

ShermanP:
But I was thinking that it might be possible to write a regular sketch in C++, or a main()-only sketch, which does everything the bootloader would do except write the data to memory, but without using any library, not even SPI. Then we could see what that looks like size-wise.

that would be the mentioned avr_boot SD bootloader