Powering Arduino with regulated 5V source over 5V pin and using USB simultaneously

Because system is not regularly supplied over USB, I want to use USB only for debugging purposes. I am using external 5V regulator which suits for my needs and I am using it to power Arduino over its 5V pin. But, I need to know is it safe to connect the USB while system is working without damaging Arduino.