Capacitor Confusion

I've got a very basic circuit using a 220μF capacitor in parallel with 5V and GND, powering an LED (see attached pic).

When I remove either the 5V and GND wire between the Arduino and the breadboard, the capacitor discharges through the LED, and it slowly fades out, as expected.

When I instead pull the serial cable and cut off the Arduino's power supply, the LED goes out instantly. Is the capacitor discharging through the Arduino (in reverse, from the Arduino's point of view)? If so, wouldn't I have to pair pretty much any capacitor I use with a diode to prevent current from flowing in reverse? That doesn't seem to be the case though, going by the tutorial circuits I've seen.

Your capacitor will be discharging thru the Arduino PCB, hence discharging quickly.

Place a forward biased diode in series with the breadboard Arduino red +5v wire to prevent.

Don't forget the essential current limiting resistor in series with the LED.

It’s there behind the blinding Krypton sun.

Understood, thank you.

Is it harmful to the Arduino to discharge through it?

Let's say I have a decoupling capacitor parallel with an IC, no diode. Normal voltage is 5V but it momentarily drops to 4.5V. Why doesn't the same thing happen? What keeps the capacitor from discharging through the voltage source, rather than the IC?

Is it harmful to the Arduino to discharge through it?

Not if the capacitor is connected to the 5V pin. Yes, if the capacitor is connected to an output pin.

What keeps the capacitor from discharging through the voltage source, rather than the IC?

Depends on the circuitry. If you open a power switch, there is no possibility of the capacitor discharging through it.