The general rule-of-thumb is use PV panels with a voltage of 1.5 x battery voltage.
In low light conditions the output voltage drops considerably, and the open circuit output voltage
spec of a panel is only for full tropical sunlight conditions.
So for 5V that means 7.5V panel (or more to allow for regulation losses).
The other approach is use a boost-buck converter with MPPT so that the voltages can be different,
and this is what a good solar charger can give you - the MPPT is going to optimize the load the
panel sees.
You could stack your panels in series groups of two, and then use a buck converter to drop down.
(remember the extra diodes to prevent reverse driving of shaded panels if you stack in series).