I think you're making a mistake of turning over the control of the servo to the varspeedservo library.
I have a bunch of servo projects and my strategy is to tell the servo which position it should be in with each pulse to the servo. This way you always know the position of the servo since you just told the servo where it should be.
As long as you don't give the servo a command to move to a position farther away than it can reach in 20ms, your position information should be correct.
I keep track of the servo's speed and position myself. I use an acceleration factor to determine how quickly the speed of the servo should change with each 20ms control cycle and move the servo's position based on the accelerating speed value. Part of the algorithm used to determine the servo's speed checks to see how far the servo is from the target position. I need to keep track of the distance to the target position so the servo will have enough time to stop using the desired acceleration.
By keeping track of the servos speed, position and distance to target, the servo can be controlled very smoothly.
I've posted code which uses this sort of constant acceleration algorithm here. I posted links to videos showing this code in action in reply #23 of the same thread. Just today I added code (untested) which moves the servos to their start positions when a button is pressed. By moving servos to their start position prior to removing power from the servos, the abrupt movement when the servos are powered on can be eliminated.
As I mentioned earlier, I've used lots of servos in several of my projects. In order control servo relatively precisely you need to control the position yourself with a 50Hz control loop. Here's a couple other examples of projects where I calculated the position of each servo at 50Hz. My 22 servo "Halloween Hex" doesn't use a constant acceleration algorithm but the program computes the IK position of the 18 leg servos each 20ms control interval. I used a bunch of different motion algorithms in this 32 servo demo. You can see how setting the position of each servo at a full 50Hz allows motion beyond the simple linear motion often seen in servo projects. These last couple projects weren't done on an Arduino but the same sort of motion is possible with an Arduino by setting the position of each servo at 50Hz. The "post #23" link given above leads to a couple Arduino servo projects showing this smooth motion.