There is a simple hardware solution…. Zero software required.
Three diodes acting as ‘steering diodes’, which allow the 0V state to close the switch, and the diodes block the ‘external’ 5V from feeding into the switch input
1N914 / 1N4148 small silicon diodes would be fine in this role.