So, the bootloader is necessary so that one can program the microcontroller directly with a USB connection, without the need for a programmer. BUT.... an external programmer can be a simple USBASP, which is nothing but an ATMegal 8/8A (or ATMegal 88). The Arduino boards already have an additional IC to deal with the USB connection - be that an FTDI IC or another ATMega (for example the 16u2).
So, why not simply have an USBASP onboard and forego the unnecessary bootloader?
some boards have it that way. with programmer on board (for example Nano Every). Zero has native USB with bootloader and a programmer too.
many boards have native USB and no additional chip (Leonardo, MKR).
for classic ATmega boards, UART FTDI USB chips were used originally and are still used on some boards (for example Nano)
So I need to reiterate the question: why waste space for a bootloader?
Well you don't have to if you don't want to. Historically it's to allow for firmware updates without any specialized hardware.
Cost.
What cost? The 16u2 could do the programming just fine without the need for a bootloader!
And the boards with CH340?
Clearly, the CH340 is more expensive than the ATMega16u2, otherwise Arduino would have used it in the newest/latest version of the UNO: https://www.arduino.cc/en/uploads/Main/Arduino_Uno_Rev3-schematic.pdf
Unless they got a really good deal the ATmega16u2 is absolutely more expensive than the CH340. We can only speculate why Arduino decided to go that route.
Really the serial bootloader thing is more historical than anything. Hardware programmers/debuggers are expensive and how they work are often somewhat guarded secrets (especially the debugging part). But a bootloader allows anyone to upgrade firmware on a device without investing in expensive hardware. Device manufacturers can also better control what programs can be uploaded and executed.
Fast forward to today. Most development boards from major silicon manufacturers (Microchip, STMicroelectronics, etc.) actually include programming and debugging functionality on the board itself, often in addition to a USB to serial converter. Arduino is the odd one out here. How come? Who knows.
Personally I would hate to see the programming pins tied up, without atleast providing debugging functionality.
so if on the board is only the MCU with native USB how do you want to upload to it without bootloader? and these are most of Arduinos today.
Nano Every doesn't have a bootloader. the SAMD11 on board with Arduino fw is a programmer.
The Arduino UNO uses an ATMega16u2 which could perfectly well implement an USBASP programmer. I hope you will agree with this clear fact. It won't provide debugging, but the way it is currently used doesn't provide debuggine either, so it's a wash.
I completely understand the reasons behind a bootloader in the Leonardo/Micro boards. There the bootloader really saves one IC. With the other boards no.
For those boards I agree, the bootloader does save 1 IC.
I wish more Arduinos did this.
I agree.
It's not a wash, it's the same pins. To use an external debugger you would have to remove the USBASP.
But in general, I think you're misunderstanding something here. I was commenting on the title of the post, "I am unsure there was a need for a bootloader after all". That's past tense, and there absolutely was a need.
If you want an Arduino compatible board with a USBASP integrated, go on ahead and design one with my blessing!
Nano Every and Uno WiFi rev 2 are the only 'new' boards without native USB.
No.
Well remove might be a bit loaded. Disconnecting would obviously be enough. Anyways, this is not the hill I'm prepared to die on, so I'm just gonna reiterate.
Fair enough.
wouldn't an always connected ICSP programmer interfere with SPI on same pins on classic ATmega?
As has been pointed out, you dont have to if you dont want to, your choice.
Most Arduinos have a Serial port USB connection which is real handy for seeing what is going on in a program, do you see it as a problem to use that same interface for programming if you choose to ?