The code is a bit confusing. I would write it so that the first thing you do is to set a variable that indicates what switch is being pressed.
Then write the code that does the action you want.
You will need a variable that notes the time when the button was first pressed using millis(), so you will need another that notes what switch was pressed the last time through the loop. I can't see any of that in your code and you need that sort of information if you want to do what you say.