Try it using the breadboard perhaps.
Be aware that many servos can't manage the whole 180 degrees. It occurs to me that the original problem may have been that the pot was asking for a position that the servo couldn't manage.
Nonetheless, if you want to make real project, I still suggest that you get better power for it.