The listed ~560mA stall current of that servo plus the ~50mA of an Uno-type Arduino is a bit much for the 500mA max rating of most USB-2 ports and the 500mA polyfuse.
The resistance of the USB lead, servo power lead, the 500mA polyfuse, and the backflow protection mosfet might soften peak current draw a bit. But that creates a dip in the 5volt supply that e.g. an analogue input or even the whole MCU might not be happy with.
A beefy cap across the servo supply pins (several thousands uF) to help the 5volt supply overcome those servo startup peaks might fix your problem.
Time to get the scope out, and monitor the 5volt supply when the servo is jittering.
Then add caps to see at what value the dips (almost) disappear.
Leo..