ATmega2560 custom board problem

You need to connect the reset pin on the mega2560 to a capacitor to the DTR or RTS pin on the USB serial adapter.

What kind of serial interface are you using?