Why did they put it in their datasheet if using it ends up limiting the PCB performance, the only way I can use that for my project is by setting up an alternate communication method and throw away USB.
Guess I will keep the external OPAMP (still weird that I seem to be the only one who faced the voltage drop issue, I will keep looking into my code and circuit cause when you are the only one you might be the problem 
)