Wow, I didn't think it would lose that much time. I wonder if you will see that change with temperature.
As for debouncing, just add reading of switches in your loop as well.
If the state of the button is normally high (say a switch on a pin with internal pullup that gets connected to ground when pressed), then take action on the first read of 0, set a flag, and wait for 5 hundreths or a tenth to elapse before you allow it to be read again.