I'm trying to make a sort of arduino games console, in which you can boot games from an external memory storage device. The bootloader seems like the most efficient way to do it
I don't think you will want or need to modify the bootloader to do what you are wanting to do, which is a good thing.
So you want to program other Arduinos from a host Arduino? Or do you just want to share the game data between them? You can send data via I2C, but I don't think you could program an Arduino through it.
If I'm understanding what you want to do, you should be able to save the game ROMs/files somewhere, and already have the Arduino(s) programmed to run them when loaded.
You can use a second board as a programmer. The sketch loaded on the board will allow you to upload a file that is stored on SD. A ready made solution can be Crossroads Standalone Programmer.
You will need a dedicated bootloader that listens on I2C at the target end. And a dedicated bootloader to transfer data over I2C to the target. You're better of trying to implement the protocol that is used for normal upload or implement the ICSP protocol.
The chances that that will work are extremely slim unless you exactly know what you're doing. Functions need to be in exactly the same place in the executable (not the source code) for that to work.
The standard bootloader does not have an option to communicate over I2C or SPI; neither to send or to receive. It can only receive a sketch via UART and write to flash.
Further, the standard bootloader does not have options to write a hex file to flash from a sketch.
You're thinking of the bootloader as if it were something like GRUB for PCs, with access to "files" and a significant user-controlled configuration.
But it's NOT like that. The Arduino bootloader is closer to the old-time bootloaders that you'd toggle in on the front panel switches. It's very tiny, and has very few features.
It has no concept of a filesystem.
It does not have any access to something like a BIOS that would provide low-level access to devices (like disks or SD cards)
It's entirely a "slave" - the Arduino bootloaders do not request code by name or any other mechanism; they are TOLD "here are some bytes, put them in memory.")
One could probably write a bootloader for an arduino (with sufficient memory) that understood SD cards and filenames and did the sort of thing that it sounds like you want, but it would be a large project (MEGA or ARM, rather than Uno/Nano), and knowing anything about the current bootloader would probably be useless. (also, it's not clear that it would be easier or more useful than having TWO arduinos - one that ran your apps, and one that had the SD card and knew how to program the first...)
there is a SD bootloader. I use it with an ATmega1284 board. (I upload the binary over network to SD card. then reset.)
For SAMD21G Arduinos there is a SDU lbrary (second stage bootloader written as Arduino sketch and prepended as HEX array for before the actual sketch).