INPUT_PULLUP will do nothing useful for a potentiometer with its ends connected to supply and ground.
If the potentiometer is more than about 5K, it will distort the readings somewhat.
The matter you describe as "the readings at the bottom end and the top end of the pot are not as linear as the rest of the readings (the ones in the middle)" is a matter of construction of the carbon track pot. It has metallisation at each end to facilitate a good connection to the carbon.
If you want more linearity, you would need to use a wire wound pot - and one manufactured deliberately for such precision.