Firstly MarkT is correct in saying that a schmitt trigger such as 74HC14 should be used for this type of switch debounce circuit. You are inputting a relatively slowly changing voltage level to the invertor which may result in false switching signals as the input slowly crosses the switching levels of the device.
Secondly, I am new here and know little of the Arduino microproc boards, so my next suggestion is a question. Is there any reason why the standard two Nand gate flip flop with a change over switch can not be used for switch debounce? This is the circuit I have almost always used for such applications.
See for example http://hyperphysics.phy-astr.gsu.edu/Hbase/electronic/setreset.html
Apologies for not attaching an image but I am still learning how to operate this site.