It's likely that you’re seeing cross-talk between the pins, especially if they’re wired together without proper isolation.
What I’d suggest is using pull-down resistors (individual for each pin) or enabling the internal pull-up resistors on the ESP32 for each button pin. That way, each button press will be isolated, and you can read the state of each pin independently.