I just got confirmation by an admin on the Adafruit board that the ground and usb pins should work.
No reply to if there is anything else in the diagram that could have caused the problem.
Such as if not using an I2C level shifting module damage the voltage converter?
I don't mind buying another board I just don't want to waste the time if I'm going to fry another one because of something in my diagram that is causing it.