I suppose I could clear that block on the port and then OR it with the new value, but its an extra step that would need to be performed each time and I would like to avoid it.
That is what you have to do in order to write any value you want. Mask AND to clear and OR to set.