There has been a problem bricking the usb chip if you supply power to the +5v bus without disconnecting the +5v bus from the usb power bus. I do not know if that was your problem since you have a custom board.
This is the schematic for the Mega 2560 R3.http://arduino.cc/en/uploads/Main/arduino-mega2560_R3-schematic.pdf
At the bottom of the schematic there is a T1 and IC7B. That is the isolation circuit. If Vin is higher than 6.6v, then T1 is disabled, and isolates the power buses. Does your board have the same circuit? If Vin is less than 6.6v, it does not isolate the buses, and has been known to damage the usb.http://arduino.cc/forum/index.php/topic,82046.0.html