If the bootloader is already on on the chip, couldn't I just pop out the current one and put in the new one?
You sure can.
That's why they put a socket on the board
(of course, if you managed to damage the '328p while it was in the board such that you'd want to replace it, whatever insult killed it will also probably take out the non-replaceable 16u2 used as a serial adapter (on genuine and faithful clones - the cheap clones use a more durable chip like CH340G which is more likely to outlive the '328p when abused), and frequently other parts involved in the power supply)
You are intending to use the Popped Out one in a project? Make sure you use a crystal/caps or a resonator with it. Don't forget 0.1uF caps on the VCC, AVCC, and Aref pins. Do not connect Aref to Vcc.
The Popped In part will need a bootloader installed if you plan to load code into it from the IDE.
Awesome. Thanks for the quick responses everyone!