So you want to drive the motor of the servo yourself, using a motor driver, and use the servo pot for position feedback?
Motors usually require more voltage and current than can be delivered by USB or a 9V battery. The servo motor may be happy with 5V, so you can use a 5V DC power supply, to power both the Arduino and the motor. You also can use a supply or battery of higher voltage,and connect it to the Arduino Vin and to the motor driver board.
I'm not sure about your idea of "twisting a pot at different speed", and how the motor should react on such input.