Noob only getting the pushbutton to work in one config but not another equal one

Hi all

Sorry if this is posted in the wrong forum, I picked the one I thought most appropriate. I am stumped by this issue and would just like to understand what is going on. What am I missing?

If I connect the pushbutton using the outer rails of the breadboard and another cable from there to the pushbutton it works.

|500x281

Yet, when I directly connect the 5V and GND cable to the button, it doesn't. The board keeps disconnecting.

|281x500

I find this especially strange as when I use the potentiometer it works without having to connect to the outer rails.

Any help would be much appreciated.