I understand that the usb port will not work, but this is not a problem, because I will only use it for programming
Than you need the bootloader. If there is no bootloader, then you can't upload via USB.
After doing a lot of searches I still cannot find the source code of the arduino micro
bootloader. (I do have an ISP)
If you still want to get rid of the bootloader, You will need an ICSP programmer like the AVRISP MKII. All you need to do is program your board with it and it will delete the bootloader.