That shouldn't make any difference.
What pin is the Joystick wiper connected to - indeed what pins are all the joystick connection on?
I just tested my code. It works fine. When the pot is in the centre nothing happens - turn it one way and the servo starts moving. Turn it back to centre and the servo stays at its last position. Turn it the other way and the servo reverses.
...R