Absolutely remove the plastic housing, the housing usually will have a "U" shaped contact in it and it likely will not come out the top of the socket.
Grab the whole header, and gently wiggle back and forth with a slight upward force. They usually will come off/ go back on quite easily.