Sigh. Let's talk about the history of the Arduino...
The original "Wiring" board was developed in 2003. There was no USBASP. The "standard" in hobbyist programmers were hacks that connected to the parallel port, or bit-banged the modem signals on an rs232 port. Usually PC-only, and by that time both of those ports were disappearing from PCs, and the OS was getting in the way of bit-banging. The board used an ATmega128 with 128k of flash, so sacrificing <8k to a bootloader didn't seem very painful.
Serial communications with a host was an important feature - one of the goals of the Arduino project was for the board to provide "physical" interaction for programs that ran on PCs. Being able to upload sketches over the same link was a nice benefit. Using a serial port drastically simplified host-side software - even now supporting "native programmers" can be a challenge across the many operating systems that it is desirable to support.
The FTDI chip was just about the only USB/Serial chip that you could get in chip form (and just barely.) I guess big vendors could get "Prolific" chips, but they mostly appeared in ready-to-use USB/RS232 adapter cables.
The Wiring board cost about $60.
The first Arduino board replaced the ATmega128 with an ATmega8, cutting the price in half (or more - suddenly the board was more home-buildable!) It had an rs232 port for communications and upload. The bootloader was reduced to 1k (12.5% of the 8k available.)
Serial ports were still going away, so USB-based Arduinos followed soon thereafter. The FTDI was still about the only available chip, so that's what was used. (it was still a pain to get through hobbyist channels.) Native-USB microcontrollers were nearly non-existent, and also hard to get. The Arduino Diecimila and Duemilanove, in the 2008 timeframe, were catching on, and clones started to appear. Atmega168 replaced Atmega8 (but the bootloader grew to 2k), and shortly thereafter came the ATmega328.
Other CPU vendors started to want in on the action, and started to offer their own "Arduino Killers" for sale at competitive prices. Most notable were the TI Launchpad(s), the NXP MBed (LPC1768), and assorted STM "Discovery" boards. Most of those came with a separate built-in debug/upload chip.
I "imagine" that sometime around then (2008), Atmel started to get pretty friendly with Arduino. "Don't you think your next board should use MORE Atmel chips? We can offer you a good deal..." they probably said. So the Uno came out with an 8u2 as the USB/Serial converter. And Optiboot, which reduced the size of the bootloader to 0.5k.
In theory, this would have been the ideal time to put programmer firmware in the 8u2 as well as USB/Serial. However:
- The existing firmware for the 8u2 is about 7k. Perhaps there wasn't room. (yeah, NOW it's a 16u2)
- AFIK, there isn't any existing code for 8u2 to do AVR ISP programming.
- Connecting an additional three signals between 8u2 and m328 might have been problematic. So far, the "interfering" connections are minimal, and have the serial comm usage as well.
- Backward compatibility! Sideways compatibility with (eg) Nano/etc. not to mention all the 3rd party boards with "other" usb/serial chips (which were finally becoming more common!)
- It wasn't needed. The goal of the 8u2 was to allow other usb drivers to be loaded, so that an Arduino board could implement a keyboard, gamepad, or midi (three popular examples.) No one saw a reason to get rid of the bootloader.
- host-side driver issues. the usb/serial port drivers were usually built in to the OS, "just worked", and were extremely likely to continue to work (despite OS churn.) Things dependent on "libusb"... riskier.
Other individual responses:
why waste space for a bootloader?
Why are you so opposed to using ~2% of the program space for a bootloader, anyway? It's a whopping 512 bytes! If your sketch is with 512 bytes of filling up your ATmega328, then you have more problems than those caused by the bootloader. Get a bigger chip.
ATmega8 running USPASP...
There was actually a group that designed an Arduino compatible using two ATmega8 chips, because that was pretty much all they could get where they lived. I can't find the discussion now. Likewise, there are boards that use ONLY an ATmega chip with USB bootloader (DIgispark and Trinket, for example.) Bigger bootloader, fewer chips. (Leonardo and Arduino Micro use a chip with HW USB.)
AFAIK, the software USB implementations cannot support "programmer" and "serial port" at the same time, ONLY support serial via violations of the USB specs (low-speed devices aren't supposed to support bulk endpoints), and are fast becoming incompatible with newer host USB controllers (having played fast and loose with the USB specs from the beginning.)
CH340 and other alternative USB/Serial Controllers.
Like I said above, the FTDI used to be the only chip-level USB/Serial converter that normal people could buy. It was a ~$5 chip (so was the ATmega8.) In fact, it's STILL a $5 chip. An 8u2 or 16u2 is a $3 chip; presumably one of the things that made it attractive at the time of the Uno design. Competitors have sprung up and have gotten better at cross-OS compatibility (OMG there were SO MANY chips for a while there that claimed to do USB/Serial for Windows Only. Sigh.) Many of them have been used in Arduino derivatives. I think Adafruit is using SiLabs cp210x, and Sparkfun is using the newer FTDI chips (all about $2.)
Boards made in China tend to use CH340 chips. It's about $0.50. But mostly made in China for Chinese manufacturers - you can't buy them through traditional "western" parts distributors, as far as I know. And it wasn't until rather recently that western folk could get such chips at all (and it might be going away
) Arduino has had in the past as strong "made in Italy" mindset. I don't think that quite survived their main manufacturer trying to steal their business, but they're still pretty far from "we're going to build the boards in the cheapest way possible."
New boards (Nano Every) have a programmer.
Yeah. SAMD11: ~$1.50. Also simplified programming interface (1 (dedicated) wire!) Also uses a 48pin main CPU to yield "about" the same number of user-available pins as the 28-pin CPU in the original Arduinos. Has voltage converters for signals (3V chip SAM, 5V ATmega4809.) Having extra pins makes it easy to re-think your design decisions.
There's also the Microchip (nee Atmel) Xplained Mini 328p eval board, which is essentially an Arduino-compatible with a full programmer/debugger chip on it. Sells for less than an Uno (but doesn't have headers.) I'm not sure what it's not more popular. (although it's presumably only cheaper than an Uno because it's subsidized as an Eval/Marketing tool for Microchip. You couldn't build your own for the same price.
)
you dont have to if you dont want to [but the 16u2 could be the programmer]
In fact, there were instructions published for how to wire (and reprogram) the 16u2 to the 328p so that you COULD use the 16u2 as a programmer. It doesn't seem to have caught on at all.
[bricking boards]
Bricking your board is surely much easier with an ISP programmer than with a bootloader.