Keep in mind one of the very important points Coding Badly made, which is that the error rate of your serial communications is determined by the relationship between the clock speed and the baud rate. Slower is not necessarily less error prone at all. The data sheet lists these tolerances, but I did this math for a boot loader I was working on, and it turned out that for an AVR clocked at 16 MHz, 38,400 was the least error prone baud rate. It was much better than the slower and faster rates.
If the temp variation causes a 4% change in clock frequency, and you are using a baud rate with say a 3% error at 1 MHz, I think you're going to have a problem.
@Coding Badly, why do you feel he is better off without a crystal?