The delays are a problem because when you're in a delay, you're not looking at the switches.
Edit: The other thing I forgot to say is, if you're not sure why a program is not doing what you expect it to, add some serial prints to get it to tell you what it is doing.