Make your life easier, and use cooperative multitasking with millis() to implement the delays in loop(). Then you can just poll the switches and all the complexity and susceptibility to programming error that comes from using two ISRs, magically goes away...