You have said two things below that are 180 deg out. You want fast or slow? If you want slow, then some simple solutions might be available. With the "knob" code and a 300 deg pot, try putting a large operator on the pot (similar to the handwheels seen on some large tv studio cams). I've hot glued a large plastic jar lid to the shaft of a pot to make it easier to make very slight changes to the pot position (similar to the servo pot adjusting tool below). If you need even finer adjustment, do the same using a 10 turn pot. If you only need only 90 deg of rotation, you might directly control the servos using $5 servo testers.
It appears that the only way I can get fast and accurate servo response is with a closed loop system.
With a closed system I can add a lot of dampening in the controller to make the movements slower - smoother