It depends on the supply and on the computer. Usually it is not a problem but as you have found, this kind of thing can happen.
If your computer is a laptop then you can unplug it from the wall and work on battery while connected to the Arduino. That will stop any strange power supply issues.