I was following this tutorial (http://www.arduino.cc/en/Tutorial/Switch) to connect a switch to Arduino Uno board, everything went great until I found that the resistor is actually connected to the GND:
Thanks in advance. 
Input pins are at an unknown, random state if they are not deliberately made to be in a high or low state. Pushing the button makes the pin high, to 5v.... but just because it's not at 5v (button not pressed) doesn't mean it will necessarily be at 0v... the pin can be picking up random signals from the air and be at any old voltage if not grounded. (Edit: and we want it to be in a known state of high or low, not lurking somewhere in between. So we connect the pin to ground to be at a known state when the button isn't pushed.)
Reason it needs a resistor, as opposed to being connected direct to ground will be clear if you look at this schematic: when the switch is closed, 5v is connected to 0v through the resistor. Without the resistor that would be a dead short and BOOM. Edit: apart from that, the 5v side "wins" when the button is pushed, since it's a direct connect to 5v, and overpowers the 0v connection (through its resistor).
Edit: Bottom line: when the button is not pushed, the pin is (as near as makes no difference) at 0v through its resistor, and when it's pushed the pin is at 5v through its direct connection.
There is a recent thread- can't find it right now- that explains the thinking behind choosing the value of the resistor.
You shouldn't need an external pull-down resistor at all; use the built-in pullup resistors instead, and just reverse your logic, I.e. wire the switch between ground and your input pin.
An open switch will read HIGH, and a closed switch will read LOW.
So, it's as if they are competing for electricity if connected this way? But why the resistor not able to pull voltage >=5V?
I'm sorry, I don't understand your question.
From what I understand from a pull-down resistor is that it pulls the voltage down to 0, so why is it not pulling 5V down to 0 as well when the button is pushed?
It is pulling the the 5v towards ground, but the circuit is a potential divider with the top resistor being nearly zero ohms.
So, the current will continue to flow through a pulldown resistor regardless the circuit is closed or opened, to "cancel" the noise that the circuit picked up along the way. Since high resistance could only allow little current to flow through it, if the circuit is closed, some current will go through the resistance while the rest goes to the input. Is that correct?
Probably best to think of voltage not current.
Have a look at the voltage divider pic here. For a pull down resistor, Z1 is a wire so ~0, and therefore Vout (the voltage at your pin) = 5 (Z2 / (~0 + Z2)) = 5V regardless of the value of Z2 as long as Z2>>Z1 (which it will be when Z1 ~ 0).
devzero:
So, the current will continue to flow through a pulldown resistor regardless the circuit is closed or opened, to "cancel" the noise that the circuit picked up along the way. Since high resistance could only allow little current to flow through it, if the circuit is closed, some current will go through the resistance while the rest goes to the input. Is that correct?
No, current only flows through the pull down when the switch is closed. When the switch is open the resistor
only connects between ground and the input pin (input pins take no current(*)) so the voltage at the pin
is also ground.
The circuit only picks up noise if the input pin is floating - in that condition the pin is very sensitive to tiny amounts
of charge and current. There is no cancelling going on.
(*) Actually they take some current, but its probably around 1000000000 times less than current flowing through the
resistor when the switch is closed, so we think of it as zero.
JimboZA:
Probably best to think of voltage not current.
Does it mean that the pull down resistor used up a bit of voltage?
MarkT:
The circuit only picks up noise if the input pin is floating - in that condition the pin is very sensitive to tiny amounts
of charge and current. There is no cancelling going on.
If no cancelling is happening, it means the resistor is there to make sure that the current to GND does not go over the rated mA (of GND)?
Thank you for being patient with me guys. As you might have guessed, I am very new to Arduino and electronics.
I strongly suggest you read up on Ohm's Law, and Kirchoff's current and voltage laws. That is not meant in any way to be snarky or mean.
Does it mean that the pull down resistor used up a bit of voltage?
No, you don't "use up" voltage.
If no cancelling is happening, it means the resistor is there to make sure that the current to GND does not go over the rated mA (of GND)?
No.
There is no "rating" for current to ground.
Have you actually read that link I posted?