dual-protocol bootloader?


I've developed a synth based on an ATMega328p. While I used my own board and my own development libs (not wiring), I'm still relying on the Arduino bootloader, and program the thing with a FTDI breakout board and avrdude. I like the development workflow I have with this. But I'd like to provide an option to also reprogram the firmware from MIDI - so people can update the firmware easily just by playing a .mid file with SysEx data.

I think I have a good understanding of what the bootloader does and how to make it implement two protocols (factor out the EEPROM / FLASH write into a function, have two loops/state machines implementing the two protocols: MIDI/SysEx and STK500). The first thing to do would be to check for a GPIO configured as an input (say #12). If it is high, skip to the STK500 loop with timeout on reads ; if it is low, skip to the MIDI SysEx loop which matches the SysEx header, read a size + a block + a checkum, verify the checksum, and write the block. A block of size 0 would mean "end of transmission, now reset". The MIDIfile would have the SysEx blocks spaced apart by a large enough time so we don't really have to care about timing/buffering.

I have no experience hacking bootloaders (or even programming a chip with AVR ISP), so before I start investing more time looking into this (and buy an ISP programmer), could bootloader gurus confirm that there could be enough space in the 2k reserved to the bootloader to cram the SysEx loading thing? If not ; is there any part of the STK500 protocol implemented in the current bootloader that could be left unimplemented to leave more room for the MIDI stuff? For example I'm willing to trade on the read function, unless it is important for something else than "backing up" a chip. What is the use case for this? Is it used by avrdude to read back what was written and check for errors?


Project: http://mutable-instruments.net/

I couldn't resist playing with the code. Two questions :

  • Why is it compiled with -O2 instead of -Os. Are there time-critical sections?
  • gethexnib, gethex and puthex are only used in the MONITOR but it appeared to me that they are included all the time. If I comment them out, when compiling for ATMega328p, I get a smaller code size (from 1924 to 1810). Maybe worth enclosing with #ifdef MONITOR...

With -Os, the gethex/puthex removal, and some tweaks (merge 0 and 1 byte response into 1 function, use loops on const buffers instead of repeated putch, factor common "read buffer length and eeprom flags" code into a function, I am down to 1372 bytes, so there's hope I'll find room for MIDI...

This will help you: Loading...
It's a modified Arduino bootloader that fits in 1024 bytes.
If you are unable to fit the MIDI code in, you could increase the bootloader reserved space on the ATMega328P to 4K.

Fiddling with gcc flags I'm now below 1k. Is this something known?

Another question:

Is it possible from non-bootloader code to jump to the bootloader. And if this is possible, is there a way to know whether we are currently in the bootloader because the device has just been powered-on (or by hardware reset) ; or just because it was triggered by software?

@Waffle: looks like I have reinvented a very similar wheel :-/

At least I've learned a thing or two in the process... 1k is super comfortable for implementing the SysEx firmware download. Awesome!

Fiddling with gcc flags I'm now below 1k. Is this something known?

What do you mean by "Is this something known?". The 1K bootloader was developed on the Arduino Developers mailing list, so the Arduino team was aware of it, but it didn't become official (I'm guessing because it would break compatibility with the extra features in the Arduino Mega). Some of the compiler flags (the ones that strip out unused functions) were incorporated to the normal Arduino sketch compilation in the IDE.

If you want to jump to the bootloader, you can use the watchdog timer to do a system reset, after which you can access the MCU Status Register (MCUSR) to detect if the reset was caused by the watchdog timer, external reset or a power-on reset. This should be suitable for you?
Read the ATMega328P datasheet for more information (sections: 8.6 Watchdog System Reset, 8.9.1 MCUSR – MCU Status Register).

Sorry, I'm new to all this and assumed that the officiel bootloader was as tight as it should be... I was not aware of this 1k version. By combining some of the things I found and the approach followed in this version, I am down to 980 bytes.

The watchdog timer approach is exactly what I want. The idea would be to jump to the MIDI / SysEx mode not from the bootloader, but from the normal firmware with a friendly procedure - like a long press on a key + displaying on screen something like "ready for OS update" - this prelude is not something I want to do from the bootloader.

All this sounds excellent, now I need to get my hands on an ISP programmer. What's the cheapest thing with good drivers on OS X?

I'm afraid you'll have to ask someone else about ISP programmers, especially for OS X.

I used this technique on Windows: http://www.geocities.jp/arduino_diecimila/bootloader/index_en.html
There is a Google Translated link on that page for instructions on compiling the software for OS X, but it may just end up being easier to get a real ISP programmer.

Thanks for the link! Maybe i'll get this to work in Parallels...

I've build this isp programmer. Standard it's not in the arduino IDE but it's easy to add it.

Ok, I've got something that looks good on code... In my firmware, a long press on 2 keys shows a "ready for os update" text on the LCD, and soft-resets with the watchdog timer. The bootloader checks for the reset type or for a pin being low and jumps to the MIDI loop, otherwise it uses the default STK500 protocol loop.

The MIDI loop parses SysEx messages containing 256 nibbles of data (1 page on the ATMega328p). I'll hack a hex->sysex converter with the right pause/timing between each block.

It seems to me that the main difference between the 1k bootloader and the original one is that the 1k version assumes all the pages written to flash always come in order, and in blocks equal to the device Flash page size. Is it a reasonable assumption?

I have bought an AVR ISP mkII, the basic Atmel thing from Farnell. I've seen many nifty programmer kits but I really wanted to try this ASAP...

Looks like it will be a fun week-end hacking!

It seems to me that the main difference between the 1k bootloader and the original one is that the 1k version assumes all the pages written to flash always come in order, and in blocks equal to the device Flash page size. Is it a reasonable assumption?

Yes that's how it is.
Although I don't think it was intentional. It should have used the address sent in the 'U' command. Maybe it was overlooked or wouldn't fit in 1K. Regardless, it seems to be working fine with Arduino's avrdude uploader.

Allowing writes of blocks overlapping several pages seems to require more logic - the big chunk of assembly in the original bootloader handles that... If I put back this chunk in place of the block-aligned, C code, I get a size of 1022 bytes for the ATMega328p, but for the other devices (those with several UARTs, don't remember the reference) it crosses the 1k boundary.

An intermediate solution would be to assume the write address is a multiple of the page size.

If someone wants to have a look or is interested in this, here's the code:

Not to minimize your effort in any way, but I fail to see why you want to implement a new "uploader" on the AtMega for the reasons you give. Is it not more practical to handle protocol conversions/formatting off board? Ater all - whatever "loader" you write - it can not do much more than write to flash/epprom and STK500 can do both already.

If you're concerned with transparency, ease of use for end-users, there is also the possibility of using virtual COM ports (aka on Windows) that reads "whatever-protocol" on input and spits out STK500 on the other side. So rather than counting bytes you will have GB's available for your protocol implementation while having the Arduino team maintain "your" chip uploader.

Hi Ben,

I want people to be able to update the firmware of my synth (or try modded firmwares) with no other hardware or software than what they already have, and it is more reasonable to assume that people making music have MIDI interfaces and sequencers installed on their computer rather than FTDI cables/breakout boards, AVRdude and virtual COM port software (I am not even sure such a thing exists for OS X).

Indeed, no matter how hard I think, I can't think of a less hassle-free, universal, way of moving raw data from a computer to a MIDI instrument than playing back a MIDIfile containing SysEx.

Also, the Arduino bootloader defaults to 57600 bauds for the ATMega328p, and MIDI interfaces communicate at 31250 bauds (maybe some of them can adjust their baud rate but this is clearly not something I would expect to find on all MIDI interfaces), so I will have to hack the bootloader anyway to allow it to support the two rates.

A last point: STK500 is a two-way protocol (the receiver needs to ack the reception of the data, send identification/signatures bytes, etc). While my current project have MIDI ins and outs that could eventually allow it to tunnel STK500 through MIDI, some of the other things I am working on only have MIDI-in, which exclude STK500.

Apparently you gave this some thought and then I trust it is all well. :slight_smile:

My assumption was that you provided MIDI through onboard USB/serial. lf this was the case you would have the option to insert whatever protocol you need between host and board - such as transparently converting from MIDI to STK500. Expecting then that this would be a lesser challenge than supporting two protocols within the limitations of the ATmega bootloader section. Crossplatform (multiple OS support) would still be an issue though.

If your targeted users require uni-directional MIDI ports however I'm starting to sense your limitations. Then again you could perhaps do-away with the STK500 bootloader and use MIDI only. For those who need to program through STK500 you could consider a tunnel (MIDI to STK500), add an SPI header to your design or perhaps provide an "avrdude.exe" that talks directly to MIDI.

Hi Ben,

There'll definitely be an ISP header in the next designs - indeed I was not familiar with this method until I had to start playing with writing the bootloader section - but it is very convenient and the barrier to entry is just a tiny bit higher than with a serial connection...

I considered ditching STK500, but MIDI is slower (7 bits only and 31250 bauds), and STK500 support is useful for using the Arduino IDE for debugging things (sometimes I write and upload quick'n' dirty sketches to troubleshoot or test one aspect of the hardware). Unless there's a way of using ISP from the Arduino IDE? But the proto I am playing with at the moment doesn't have an ISP header anyway...

A last thing: the main firmware of the synth is a large exercise of byte-counting (and to some extent, my professional work is), so I already knew how to approach the question for the bootloader :slight_smile:

Unless there's a way of using ISP from the Arduino IDE?

You can redirect the Arduino IDE to using an ISP programmer through an override in the boards.txt file - so this is pretty straight forward.