You are not using the internal pullup resistors. So, you need to have external pull-up or pull-down resistors.
How are the switches wired?
Your problem sounds like a floating pin condition.
Even if the switches are wired correctly, the only time that the pins are read is after the delay. That requires that you be holding the switch down when the delay ends. Are you?