Ground. Which is to say - the OTHER lead on the button goes to ground.
It might seem a little odd that "low" becomes "on", but it means that you can connect the other side of the button to the chassis of your build, and you only need the one wire going from the button to the arduino. If it was the other way around, there's have to be a wire carrying 5v snaking in and out to reach every button.