@PaulS:
The OP said
What I would like to do is turn one LED on with a button press and then on the second button press turn the first LED off and turn the second LED on.
and that's what the code does. If the OP wants to handle the case where someone holds the button down then they could replace the "if" statement with a "while" statement.
I thought that the code was documented enough to show how it was wired, but I'll explain how I set it up on a breadboard. A momentary button was connected on one side to 5V and the other to ground through a 10K pull-down resistor and to Arduino digital pin 3 ("buttonPin") through a 100 Ohm resistor. Two LEDs were connected to ground via 330 Ohm resistors and their anodes were connected to Arduino digital pins 9 and 10 ("oddLED" and "evenLED").
When the sketch runs neither LED is lit. When the button is clicked the LED connected to Arduino digital pin 9 ("oddLED") lights. When the button is clicked again the lit LED turns off and the LED connected to Arduino digital pin 10 ("evenLED") lights.