Still having issues using ArduinoISP sketch to install bootloader

Other Hardware 3rd Party Boards
2

To program the ATMega2560 using an Arduino as an ISP you'll need to do a bit of fiddling.
You'll also typically want to to use the 'Arduino as ISP', rather than 'ArduinoISP'.
Can you post a schematic of your custom board?