They can oscillate because they are high gain ampliers. If you set things up to satisfy Nyquist's criterion,
it oscillates. The capacitors affect the poles and zeroes of the feedback loop network. When a regulator
oscillates you typically take out all the chips it is supposed to be powering - seen it, been there, replaced
all the components with a hot-air rework station. (Though that was caused by accidental parallelling of
two regulator outputs, but its the same basic issue, regulator sees the wrong impedance at the output).