Should I just delay a little in the setup loop in the host side driver?
Yes. Two weeks seems like a good amount.
Meanwhile, you might consider reading the sticky threads at the top of the forum that you were supposed to read BEFORE posting here. I'm sure something will occur to you by the time you are done.