I have designed a simple PCB circuit board that is powered by either the 5V from a USB, or if a battery is plugged in, powered from a 3.3V and 5V regulator. I am no expert in electronics, but have attached my schematic below.

I realized that if I remove and short D1 from the circuit and have it powered by USB, the 5V power from the USB will travel backwards through the 5V regulator and power the 3.3V regulator. (With no battery plugged in).
The regulators I am using are the LM1085ISX and the data sheet is attached.
Datasheet On page 12 of the datasheet it says that is has protection diodes that should prevent this reverse voltage, so I am a little confused as to why this is happening.
I know this probably isn't the ideal situation, but is it okay to have this reverse voltage flowing through the regulator, could it possibly damage it?