using either a rotory encoder or button press count to change pwm mode

I'm not too interested in the details of wiring your rotary switch up to the input pins, but assuming you get that right it will be possible to infer the switch position from the inputs. Given that you will know how the input values relate to the switch position, it will not be hard to implement the code to read the inputs and determine the switch position. The other behaviour you describe also sounds straight forward. Which part are you having trouble with?