•The 8U2 holds a bootloader. I load this on through a serial interface via some pin headers before handing over the device.
The 8u2 is a processor, just like the ATMega328 (or whatever your main chip is). It's got it's own code (it has 8K of flash) which also has a bootloader to get code on to it. The code it runs is to make it look like a USB serial device to the host computer, and relay those bits via some pins to the ATMega328.
So you're really dealing with two chips with two bootloaders and two user programs. I think this is what you mean, but just want to make sure.
•Once someone else takes possession they can plug it into their computer via USB, it is recognized as "My Custom Arduino Device" (how to do this?)
Are they on Windows? If so all you'll need to do is modify the driver .INF. Check out this page:
It describes how SparkFun wired up & configured their version of what you want to do, and has a link to their drive .INF. Open that file up in a text editor and near the bottom you'll see
DESCRIPTION="SparkFun COM Port"
Change that to "monkey123's custom Arduino board" and away you go.
Should I write my own bootloader or are there good open source ones available?
If you mean the bootloader for the main chip (ATMega328) then the one that everybody (even the Arduino team with the Uno) uses these days is Optiboot:
If you mean the bootloader for the 8u2 then here's the code the Arduino team uses: