Even if I don't push the button, it reads HIGH value

You are taking a big risk. Using the blue/black bus line of the breadboard for 5V is an accident waiting to happen. Always use the blue/black bus for ground. Maybe use the red line on one side of the BB for 5V and the red line on the other side of the BB for 3.3V. This won't prevent all mistakes, if course, but it should significantly reduce the risk.

Here is something to check: often the bus lines on BBs can have a break in the middle which you need to connect across with wire. Normally the break is indicated by a gap in the red & blue/black lines also.