- I connect the arduino power supply, everything seems fine
- I wait for a few minutes just in case and it doesn't matter how much time I wait, it never starts
Almost assuredly this is a voltage issue with your board drawing enough current to lower the voltage to the uC ...
When you plug in the USB, you are supplying USB current in addition to Arduino PS current. The voltage goes up beyond "brown out" and things start. Removing USB lowers the voltage but not quiet enough to force the uC into brown-out.
Ray