(updated) I have a 16mhz crystal on board with the two capacitors. I can upload sketches fine via a USBtinyISP but the speed is slow. I've tried using Nick Gammon's fuse calculator and board uploader on a stock 2560 board but both hang when ICSP is connected to the custom board. I've also read to switch the timing in Nick's scripts to CLKDIV128 but that hasn't helped.
I can communicate with the custom board via avrdude and the tinyISP.
avrdude -B 25 -c usbtiny -p m2560 -u -U hfuse:w:0xD8:m
avrdude -B 25 -c usbtiny -p m2560 -u -U efuse:w:0xFD:m
avrdude -B 25 -c usbtiny -p m2560 -u -U lfuse:w:0xFF:m
Does this look ok?