Use a schottky diode, voltage loss reduced to ~0.3V range. Only get 500mA from the port, pick a diode accordingly.