You are probably just being paranoid and, the more you plug/unplug, the easier it is likely to get. If I understand you correctly, the Xbee module has to go into another device in order to be programmed, and I guess you only do that once anyway.
I have noticed variation int the way a shield installs. I have wondered if some have harder brass in the pins and it takes a bit more to straighten them.