The point is the series inductance to the electrolytic makes it much less effective at high frequencies - it won't be as "stiff" in controlling the voltage on short timescales (its fine in the longer term).
Close, but not exactly there.
Take a typical capacitor model (C+ESR+ESL). Model the line from the power source to your load as a 0ohm resistor + inductance. Now, put a sudden load on the whole thing, and watch the i-t response.
Play with the ESR+ESL combination and you will see why people parallel small (highspeed) capacitors with large capacitors.
Here is a hint: the wire inductance functions as a high-impedance element for highspeed loads (aka a digital circuit). Without a local decoupling capacitor, you will see substantial voltage drop on the load as the current goes up suddenly (di/dt for an inductor).