It seems the above sketch's Serial output got faster then 500ms when the sketch ran for a while, and the servo movement is a bit sloppy afterwards.
Why not print out the servoCurrentMillis() value. Then, you won't have to use words like "seems to". You can use "does" (or "does not") instead.