I know that Arduino mega is programmed by serial 0 since the usb to serial converter is attached to it. but i want to use an external USB to TTL converter to program it through Serial 1. What changes do i need to make in the bootloader?
I know there is Optiboot compiled for ATmega2560 and Serial1 here:
https://github.com/MCUdude/MegaCore/tree/master/avr/bootloaders/optiboot_flash/atmega2560
MegaCore doesn't have a menu option to select the serial port but you could simply edit boards.txt and change:
2560.menu.clock.16MHz_external.upload.port=UART0
to:
2560.menu.clock.16MHz_external.upload.port=UART1
Of course if you want to compile Optiboot yourself for UART1 it's easy enough. The source is there in MegaCore or you can get the ATmega2560 version from the original source here:
As for the stock bootloader, it's probably about as easy but I don't have any specific knowledge of that.