Yes, if you go below 3.2V a cell, you'll damage the batteries. So buck only required.
The Dollar Tree stores here carry 12V to 5V USB chargers. Has everything you need, although you may need to adjust a couple of components. They use the MC34063 switching regulator. The inductor in the ones here are shrinkwrapped, and so rewindable. There are many online calculators to help you with the right values.
This one lets you input a voltage and a range as a percent, plus or minus.http://dics.voicecontrol.ro/tutorials/mc34063/
6.4 to 11.8V would be 9.1, +-30%.
A 7805 will not do, as it requires a minimum of 8V to regulate the output to 5V. A shame to waste half the power, in any case.