Solved: Serial Port Connection Time on Arduino 101

You could also try tweaking the USB power settings too. All min are set to save no power at all and keep everything up and running.
Also no sleep settings at all.
USB 3.0 can sometimes be a PITA with Arduinos so if you have USB 2.0 try stick to those.

Another thing to maybe be wary about is Arduino related USB drivers from other sites.
Its taken me almost two months to track down a serial issue and only then after a lot of help from some of the backroom members of the Arduino dream team. May not be in your case but USBSER.SYS in my case was at fault and was either the Adafruit one or the .ORG one.

I was unable to roll back and ended up with a bare metal restore from my server.