I don't know what you mean by "Surely not with an external pull down?"
Is my wiring wrong?
I added INPUT_PULLUP, but it didn't seem to make any difference.
Sorry, that one was for LarryD.
Pullups and Pulldowns are mutually exclusive and Larry hadn't realised there was a pulldown.
EDIT.... one other thing to check, is that you have the switch oriented correctly on the board. I can't see the legs in the photo, but those square, 4leg switches are often 90 degrees out. Look at the pic I attached of a typical switch and see if you have the switching correct. Might be an idea to get your ohm-meter out and satisfy yourself which pins are always connected and which are switched.