dhenry:
If the capacitor value is too high the the high frequency responce suffers. That is why you have to use both a small cap 10 to 100 nF in parallel with a big one 10 to 100uF.
If too high capacitance causes frequency responce to suffer, why would adding more capacitance helps?
It is for something else.
The point is the series inductance to the electrolytic makes it much less effective at high frequencies - it won't be as "stiff" in controlling the voltage on short timescales (its fine in the longer term). Small ceramic placed close to the chip is there to do the initial decoupling after the step-change that would otherwise knock the supply rail down (or up). The combination covers all the timescales. The voltage regulator takes over at lower frequencies.
For many digital circuits you can probably get away with being a bit sloppy on decoupling, but if you mix analog and digital you immediately have to be much more systematic on decoupling to reduce noise injection via the supplies. The larger the currents being switched (LED arrays for instance) the more careful you have to be to decouple adequately. Follow best practice always and you don't have to worry though 