You need to update your previousMillis variable when you detect the button has been released, your current code does not do this.
Obviously you only want to update that variable once, when the state changes from on to off. I would look into a state machine as advised above.