No, the problem is the inverter! It must be a schmitt-trigger input. a 7414 not a 7404 for instance.
If you were using CMOS you could add hysteresis with a resistor feedback network, but that's tricky with TTL, I think that 74AC04 must go. Try a 74HC14 then you can use a much smaller capacitor (0.1uF and a 100k pull-up for instance gives 10ms time-constant)
The existing 10uF is being shorted into the switch each time it closes creating a tiny spark, which may eventually erode the contacts if the switch is small. Adding 100 ohm or so in series with the switch will prevent such damage. With a 0.1uF cap this damage is negligible.