Yeah, that is actually a pretty good point on just using a buck or boost converter.
Actually, you could use a buck/boost converter to supply the arduino with a constant comfortable voltage, while running servos straight off batteries....
No endorsement of those specific vendors - there's a huge number selling same units and similar ones.