Thanks for all the replies.
I was just able to test this with a variable supply, though I did only test one servo.
I used a 1 second sweep from 45 to 135 and back continuosly while slowly lowering the voltage. As expected the servo slowed down more and more but still seemed to work all the way down to ~3.0V at which point it started to flutter back and forth (+-10 degres or so) quickly. Increasing voltage back to >3.0V restored it back to the sweeping movement.
So my conclusion is that, even with some spread, there should be no problem running the servo of one LiPo cell (3.7V).