When connected to the Arduino 5V supply, voltage dropped to 4.8V, which made me wonder about voltage going into the motor.
That is your problem.
NEVER power servos or motors from the Arduino power supply OR the 5V output. They inject noise, overload the voltage regulator and can actually destroy the Arduino.
Always use a separate power supply and connect the grounds. 4xAA batteries will work fine for one servo.