I have boiled my code down to a vague area and can re-create the issue. (Still getting it down to specific code)
Everytime I upload this sketch it causes the issue listed above, and uploading the Blink sketch fixes it again.
I'll try it with a hub inbetween later when I investigate this some more. (and find specific code which is causing this)