Buck-Boost troubles: 5v output becomes 20v+ if input drops below 3.5v

It's frustrating: the buck regulators work fine in low voltage situations (e.g voltage just drops off really quickly). However, the 5v output voltage starts to drop off around at around 6v. So using this reduces my margin for voltage drop if all the servos are active even further. Using the buck-boost will at least let me drop to 5v, but with potentially catastrophic consequences if it drops below this.