A few times this has resulted in a major OSX error saying that I might shutdown/restart my computer
You really shouldn't see a stop message when disconnecting the Arduino. Which version of OS X do you have?
Have you installed the latest drivers from http://ftdichip.com/Drivers/VCP.htm ?