Yes a boost switching voltage regulator is the nicest solution. There are tons of very inexpensive asian items that would fill your need. I've bought a couple of such boost and buck (for reducing voltage rather then increasing voltage). Being switching regulators they are much more efficient which is important when building battery powered projects. Here is just one example:
http://www.ebay.com/itm/3-30V-to-4-35V-DC-DC-Boost-Power-Supply-Board-Step-Up-Adjustable-Module-LM2577-/370669720641?pt=LH_DefaultDomain_0&hash=item564da31441
Lefty