Program works only with USB and not direct connection

Without knowing what and how you connected to the Arduino it is hard to tell what the problem might be. Looks like you drawing a lot of power of the Arduino to power something else!?