I think you are on the right track with the states, but you do not have enough of them for easy debugging.
How about these names for states (you may think of others for your application)
IDLE, OPENING, FULLY_UP, CLOSING, FULLY_DOWN.
Now try writing down what you want it to do for each state transition, and what will start the transition. Try and cover
the cases from all sensor inputs that are available, including your buttons. That will give the logic of what happens at
each state, and should make it easy to code. As part of what will happen, decide on which next state is wanted.
For example:
Its IDLE, and light sensor says dark => switch to CLOSING
Its FULLY_DOWN and light sensor says dark => do nothing
Its FULLY_DOWN and light sensor says daylight => switch to OPENING
Its OPENING and not at encoder limit => motor up
Its OPENING and at encoder limit => motor off, switch to FULLY_UP
Its OPENING and down_button pressed => do nothing (or if that is the wrong spec for you, say something else).
Its OPENING and light sensor says dark => do nothing
Its OPENING and light sensor says daylight => do nothing
Then try drawing out a state diagram (optional, but really useful). In your case, for each state there are inputs from the sensor, encoder and buttons.
Don't try "clever" tricks like state = state -1, explicitly set the next state wanted, e.g. state = OPENING;
(A useful, but advanced idea is a C++ enum).