I'm aiming on blinking the LED while continuously controlling the servo.
In loop(), test whether it is time to change the state of the LED using millis() timing. If it is then do it.
Whether or not you just changed the state of the LED, read the pot and write to the servo.
Two separate sets of actions.
Yes, changing the state of the LED will delay the pot read/servo move but loop() repeats thousands of times per second so you are not going to notice.