Serial port 'COM19' already in use

I've posted it in another thread but realized that the problem I encountered is different from the thread starter. Pardon me transferring it here. Hope someone can help.

I realised one thing while trying to figure out the port already in use problem. After getting it to work following the cumbersome way of disabling, rebooting and enabling the serial port, I realised that it may be the ide refusing to release it hold on the port (pardon me if I am technically incorrect with terms and jargons here).
When I quite the IDE and started it again, the uploading works ok. Once I plugged out the USB cable and plug it back again, the problem starts. This occurs regardless of whether I unplug the usb cable with IDE running or after I shutdown the IDE.
So it seems to me that the device still hanging on to the serial port is the IDE than any other program.
Anyone has any solution or things I can try with?