Are you measuring your 5V while the cap is connected to the charging supply? If you are connecting a fixed voltage current limited 5V power supply directly to the capacitor you're bound to see 5V while it's charging.
You'll only know anything useful about the state of charge when you measure the capacitor on its own, i.e. not connected to anything else.
Steve