Exactly! As long as "other stuff" does not use delay() the main loop can be very responsive to switch presses.
Therefore dhenry's suggestion:
delay(LOOP_DLY); //delay the loop a little bit
... is not necessary. No reason to throw in arbitrary delays.