I suspect that powr is consumed by other devices connected to the same 5 volt USB bus or other 5 volt loads inside the computer box. If USB power really is derived from from the 5 volt computer power supply, there is a lot of load connected to the power bus. Clearly the USB power lead is not diode isolated because 5 volts is available at the end of the USB cable. The USB current limit is what? half an amp? but some things come with the cable that splits and use 2 ea half amp allocations. I don't think polyfuses notice which way current travels. If there is some sort of active current sense device metering power out to a USB port and protecting if it from over current, does it know what to do if the current reverses?
I guarantee you that an external 5.2 volt regulated power source will supply current to a USB port and it will be a design consideration for me in the future.
JC