It looks like the servo library uses timer interrupts to pulse the servo, which would be delayed if serial interrupts fired.
It certainly should be possible to do the PWM in hardware (using the timers directly) rather than via interrupts, then it would not be affected by serial comms. Probably someone has already done this, but I don't have a link to hand.