did you configure both A4 and A5 as input first?
It does not matter what the pullup on your board is. what matters is the cpu internal pullup is to 5v and your sensor board pullup is to 3.3v and you don't want to mix them. A4 and A5 are special pins that are open collector specifically designed so if you disable the internal pull up, only the external pullup wll be used. its all in the atmega328 datasheet if you want to read about it.