Why Is a Bootloader Necessary?

I have two devices that use the ATmega328P: Arduino Uno and a Gertboard. The Gertboard connects to my Raspberry Pi and programs the ATmega directly over the SPI bus. For new Atmegas, I run a simple program that uses avrdude to program the fuse bits - that's all it does. Then, I can download a sketch using "upload using programmer" from the Arduino IDE. There is no bootloader installed on the ATmega328P. The Raspberry Pi becomes an AVR programmer.

For the Arduino, besides programming the fuse bits, is it true that the only other purpose of the bootloader is to program over RX and TX rather than SPI?

Since SPI does not require a bootloader, why can't an Arduino be programmed over SPI directly? I see there are several USB to SPI chips and, of course, there are the AVR programmers. Why can't an Arduino have USB to SPI circuitry, built in, rather than USB to RX/TX? Couldn't the ATmega16U2 be connected and programmed to use SPI?

I'm just trying to understand something that has me baffled. - Thanks

If you use a virtual com port FTDI/USB device it can program the UNO if there is a bootloader on the UNO.

If there is no bootloader on the UNO you program the UNO with the ICSP connector (SPI as you say).

For example see what Nick Gammon has done on his web site:
examples

MartyL:
besides programming the fuse bits, is it true that the only other purpose of the bootloader is to program over RX and TX

fuse bits are not one of the purposes of bootloaders. bootloaders do not have this ability, only isp.

for nearly all my projects a $2 usbasp does the job. for those few cases where pinout prevents this (ie relay or tiny) i use a single wire custom bootloader but very rare. aside from hogging resources a bootloader only adds another level of complexity and failure. it did a great job of corralling captive market for the arduino company though.

MartyL:
Since SPI does not require a bootloader, why can't an Arduino be programmed over SPI directly?

Clearly, it can, as you indicate.

MartyL:
I see there are several USB to SPI chips and, of course, there are the AVR programmers.

Certainly are.

MartyL:
Why can't an Arduino have USB to SPI circuitry, built in, rather than USB to RX/TX?

It could, but ...

MartyL:
Couldn't the ATmega16U2 be connected and programmed to use SPI?

It could, but the USB interface is not as standardised. The USB to serial (TTL) interface is much simpler and provides a convenient connection to various software as standard ("virtual") serial COM ports. It effectively performs double duty as both programming and versatile communications interface. It is as simple as that.

{I suspect it is also cheaper.}

is it true that the only other purpose of the bootloader is to program over RX and TX rather than SPI?

Yep. Well, I guess it provides some protection from forms of programming that could "brick" a chip, at least theoretically.

The main reason to do this is that PCs already understand "serial ports." To implement programming over SPI, you'd have to have some sort of custom driver on the PC side, and/or some kind of usb to SPI conversion implemented on the Arduino side. Those quickly become a more complex thing to implement and deploy, compared to a "simple" serial port. Especially in the context and timeframe of the original Arduinos (the original Arduino connected to an actual physical serial port. And it is a pretty big advantage to be able to use different hardware (rs232, ftdi chip, ftdi cable, random other usb/serial cables, 8u2, 16u2, pl2303, cp2101, ch340...) in between the PC and the AVR...

westfw:
and/or some kind of usb to SPI conversion implemented on the Arduino side.

MartyL was correct in pointing out that there are USP to SPI converter chips just as there are USB to TTL chips - I was careful to review the situation before replying.

So it would strictly speaking, be just as easy and no more complex to incorporate a USP to SPI converter chip and use one of its auxiliary control lines to put the ATmega into reset.

It is just more convenient and versatile overall - and I imagine, cheaper - to "speak" in asynchronous serial.

Thank you all for your answers I have a much better understanding of bootloaders.

Of course, John1993, is correct saying that the bootloader does not program the fuse bits. The program that loads the bootloader must do that.

there are USP to SPI converter chips

There are now. When the Arduino usb boards first came out, even the FTDI USB/Serial chips were difficult to find at "intermediate" distributers (digikey/mouser/etc.) And I don't know if any of the USB/SPI chips are supported by AVR programmer software, even now. (All things considered, it's a little bit mind-boggling how much the USB-capable CHIP market has changed in the last 5 to 10 years. In ~2005, you had your choice of several $30 USB/RS232 adapter cables from the likes of IOGear and MacNally (mostly for connecting PDAs and other rs232 devices to Macs, because PCs still had real serial ports.) And there was one vendor selling the FTDI chips, and one vendor selling microcontrollers with (low-speed) USB capability... (ok, you had more choices if you were a large company able to negotiate with chip vendors that didn't sell through distribution...))