Yes, just to emphasize - it's the crystal/resonator frequency difference that requires a different bootloader, not the voltage difference per se. So if you have the right bootloader for a 3.3V 8MHz Pro Mini, you can run it all day long at 5V with no problem. However, running a 5V 16MHz Pro Mini at 3.3V does not work, but only because, as Wesstun says, the 328P won't run reliably at 16 MHz on a 3.3V supply.
There may be other things that have to be adjusted, but I think the main difference is that if the bootloader is doing UART transfers at, say, 9600 baud, you need a different prescaler value for an 8MHz crystal versus a 16 MHz crystal.