Boosters are not bad just what you would call inefficient.
Your project uses say 1 amp at 5 volts, if you drop down you will most likely still only use one amp but when you Boost you trade current for volts so you use more current to get the same 1 amp at 5 volts with another circuit in between that also uses current.
E.g. You start with 6 volts, boost it to 9 volts. Thats a 1/3 increase so assume a 1/3 increase in current so 1 amp becomes 1.34 amps. Then allow for the booster to be 95% efficient meaning you loose 5%.
So add 5% to 1.34 amps.
You now draw 1.41 amps to get your 1 amp at 5 volts. Not taking into account the current to run your boost and buck circuits.
But some times you just have to go this way.
Daz