Different Servo situation, but same odd jitters

I bet your 2 libraries, Servo.h and SoftwareSerial.h, are in conflict over the timer.
You might try Servo2.h instead.