Good detective work.
There is a description of the internal pull-up resistors here
so when using pinMode to set the same pin mode for both OUTPUT and INPUT, the pull-ups can be disconnected (and should be if you don't want them to be active)
pinMode(pin, INPUT); // set pin to input
digitalWrite(pin, LOW); // turn on pullup resistors