Port Busy Error when trying to upload or open serial ports on a custom pcb

Hi, I've had some PCB's made with ATMega32U4's on them, which have had arduino bootloaders put on them. They're relatively simple designs with just a few sensors and a CAN interface connected by I2C and SPI respectively. The board shows up as an Arduino Leonardo when I plug it in.

The issue I am having is I can't upload anything to them at the moment, and I cant open the serial port to see if it's working EXCEPT on first boot of my PC. So the first time I open the port via arduino monitor I can see the test code working BUT if I unplug and re-plug the part board back in then it says port busy. When I try to upload anything it compiles fine but then gets stuck at the "Uploading..." phase.

So far I have tried using different ports and using a different IDE, plus trying with a different board, but always the same errors. I've also tried different code, and just to be safe made sure I could upload the same code to an arduino Nano.

Sketch uses 6118 bytes (21%) of program storage space. Maximum is 28672 bytes.
Global variables use 485 bytes (18%) of dynamic memory, leaving 2075 bytes for local variables. Maximum is 2560 bytes.
Forcing reset using 1200bps open/close on port COM21
processing.app.debug.RunnerException
    at cc.arduino.packages.uploaders.SerialUploader.uploadUsingPreferences(SerialUploader.java:152)
    at cc.arduino.UploaderUtils.upload(UploaderUtils.java:77)
    at processing.app.SketchController.upload(SketchController.java:732)
    at processing.app.SketchController.exportApplet(SketchController.java:703)
    at processing.app.Editor$UploadHandler.run(Editor.java:2055)
    at java.lang.Thread.run(Thread.java:748)
Caused by: processing.app.SerialException: Error touching serial port 'COM21'.
    at processing.app.Serial.touchForCDCReset(Serial.java:107)
    at cc.arduino.packages.uploaders.SerialUploader.uploadUsingPreferences(SerialUploader.java:136)
    ... 5 more
Caused by: jssc.SerialPortException: Port name - COM21; Method name - openPort(); Exception type - Port busy.
    at jssc.SerialPort.openPort(SerialPort.java:164)
    at processing.app.Serial.touchForCDCReset(Serial.java:101)
    ... 6 more