Here's my PCB layout. It doesn't do much good now since it's already been made. Sorry, we switched to the LSM9DS1, but nevertheless, the schematic still remains the same, and we're using Adafruit's module. Right, looking at the schematic for Adafruit's module, both 3.3V and 5V have pull-ups. If powering from 5-volt, Adafruit has a regulator which steps down the 5V to 3.3V, so you can safely power the module with 5V or 3.3V. In this case, R1-R4, external to the Portenta and Adafruit sensor, screw everything up.
