Put some serial prints in to check what actual values you're writing to the servo. Since servo.write() takes values from 0 to 180 your calculations look odd to me.
And it doesn't look as if you are catering for +/- buttons being held down which normally sends a repeat code not the standard button code.
Steve