Do you have *another* capacitor? You can put them in series which will accept 10V (albeit for half the effective capacitance).
Yes, you could disconnect the 9V source when the voltage reaches 5V, but this sounds relatively complicated.
You could put the 9V through a voltage divider to get no higher than 5V, though this wastes current and slows down capacitor charging (any more details on what you're trying to do?)
You could also put a 4.7V zener diode across the capacitor to limit the voltage. Again, the best solution is going to depend upon what you're actually trying to do.
Check out our new shield: http://www.ruggedcircuits.com/html/gadget_shield.html