The reason for the two capacitors is that they stop the 7805 oscillating!
A voltage regulator contains a high-gain amplifier (rather like an op-amp) as well as a voltage reference and a power output transistor. If you leave out the capacitors, it may well work, but there's a risk of very high frequency oscillations. These oscillations will get transferred out on the power wires to your circuits, which will then misbehave in strange ways.
Note that even a battery will not produce a stable volatge if the current drawn from it varies (look up "internal resistance"). So you'll usually find a capacitor across the power supply of a circuit, even if the supply is not coming from rectified AC.