I don't know what 'pretty basic devices' means in electrical terms.
If you power the Arduino from a 5V supply via the USB connected then you will be bypassing the onboard regulator and any noise in the supply will go straight into the microcontroller.