STK500 protocol.
You have the bootloader sources. Read them.
You can turn on verbose logging of the upload in the Arduino preferences. That will log the step-by-step process of uploading.
You can read about the 'avrdude' program that does the uploading.