It is absolutely no problem, but once if the chip is set to use external clock or crystal, it (clock signal generally) must be present otherwise the MCU will not start. So, just replace the crystal. Of course, the bootloader have to be changed for exact frequency.
However I'm not recommending to use 32MHz. I have some experience with overclocking on ATmega1284P. Some features doesn't work at higher frequency as 25MHz, e.g. prescaler for ADC. I had the best results with 24MHz. The MCU works reliably at 24MHz, no getting a heat and all features seem to work.
...but 328P can be different story!