You cannot just replace the word "delay" with the word "millis" and have your program work. I suggest that you read the documentation for the function millis at millis() - Arduino Reference
and also read Using millis() for timing. A beginners guide
Almost nothing else can occur during delay ("Almost nothing else" means "Nothing except interrupts"). However, if your loop function will check the input pin, then there can be a response nearly as fast (or perhaps even faster than) as using an ISR, and much easier to debug. You would, indeed, be using an if/else instead.
Please revise your program after you read both links that I posted above. Using millis() instead of delay(...) usually means restructuring your program, but your program is small enough that any restructuring should go quickly.
I cannot say how fast this can go (because I do not know all Arduinos and you have not told me what you have or what its clock speed is) but it can go very fast. Microseconds, even.
This code
pinState = (PINB >> READPIN) & 1; //PINB is the register to read the state of the pins
costs time on many Arduinos because of the shift operator ">>". Because you only compare to zero, it would be better to shift the "1" in the other direction and not shift the value from PINB at all. The compiler should be "smart" enough to calculate this constant once and use it instead of a shift. If the code is sufficient as it stands, perhaps you do not "...need the loop to be able to be interrupted at any time as quickly as possible...".
For some people, "...as quickly as possible..." is femtoseconds. For other people, "...as quickly as possible..." means seconds. What is your requirement?