I think the ATmega2560 bootloader uses the stk500v2 protocol, not the stk500 (v1) protocol used by earlier Arduino models.
If you hold down the shift key when you click on the Upload button for an Arduino sketch it will show you the commands it uses to upload the .hex of the sketch. You can than adapt that command to your own .hex file.