I have no idea of the spec of the switch but I've tested it with my Ohmmeter and worked out I need to use 3 of the pins on one side: Power to the center pole, "off" state (left pin) to 10k resistor and then back to ground and "on" state (right pin) to pin 2.
OK. Just make sure the "center" terminal is the
electrical center. (It usually is.)
Then +5V can be connected to one of the other terminals, and ground to the remaining terminal. That way, the center is either connected to +5V or ground, depending on the switch position. In that configuration, you shouldn't need the 10K resistor at all.
But, the 10k resistor (between 5V and the switch) is slightly "safer" because there are some very-rare switches (called "make before break") where the two "other" terminals are connected together for a moment in-between positions. If that happens, you'll momentarily short-out the 5V. Everything will
probably survive a momentary short, but the Arduino would reset/restart and that would be a problem.
You can test the switch (with or without the resistor) with your multi-meter before connecting it to the Arduino. Just check for 5V (at the center) in one position and zero volts in the other position. And, it would be a good idea to turn-off the power and use the ohmmeter function to confirm that the "zero volts" connection is a good-ground, and not simply an open.
The REAL solution is to use a pull-up resistor: The Arduino has optional internal pull-up resistors which can be enabled (see
this page). The pull-up resistor pulls the input high (+5V) when there is no connection. Then,
you only need two connections to your switch to force the input low (to ground) when the switch is on.Brandeaux suggests using a pull-
down resistor. That will work fine too. But, pull-ups are the more standard way of doing it, and the built-in resistors are pull-ups.
With a pull-up resistor, you are "shorting-out" the input and shorting one side of the resistor to ground. That's fine, since only a teeny-tiny current flows through the resistor. If you were to make a "hard connection" to 5V (without the resistor), turning-on the switch would short-out the 5V supply, shutting everything down.
I've tried adding a delay to the loop (from 10ms to 100ms)...
That's only going to make things
worse! 
As the program is "stepping" through the loop, depending on where the program execution is in the loop, the input-read might get delayed or your "action" might get delayed 'till the delay is done.
FYI - As a general programming rule, it's best to avoid using delay(). Because while delay() is running, your program can't do anything else. (Of course, you
can use it when appropriate.) So, if you you want to delay
something without holding-up your entire program, you can read the "time" and compare like the
Blink Without delay()Example.. With this techinque, you can have multiple delays/timers running at the same time, and they won't interfere with each other or with your main loop.