On any given pass through loop, which probably happens far oftener than you think, only one button is pressed. Only one button can have changed state.
Therefore, you need to treat the buttons, and their states, as completely independent.
In the loop function, see if button 1 is being presses, but was not before. If that is the case, set ledState.
Then, see if button 2 is being pressed, but was not before. If that is the case, set colorState.
The value of colorState has nothing to do with whether button 1 is being pressed, or not. It depends on (or changes as a result of) ledState, not whether button 1 is being pressed.