slipstick:
Are you measuring your 5V while the cap is connected to the charging supply? If you are connecting a fixed voltage current limited 5V power supply directly to the capacitor you're bound to see 5V while it's charging.
Not necessarily so. With the power supply in current limit mode, the power supply will reduce it's voltage to limit the current. What should be seen, in this case, is the supply voltage gradually rising. It should take around 400 to 500 seconds. I checked the datasheet on a 4F 5.6V supercap and saw an ESR of 30Ω, so at 50mA, that's a voltage drop of 1.5V, so if the starting voltage on the cap is 2V, then the Power Supply would be at 2V + 1.5 = 3.5V
3V + 1.5V = 4.5V, so that would explain why the OP is seeing a higher voltage than one might expect. Perhaps the ESR on the OP's supercap is higher than 30Ω.