Other folk have had the same problem. See here where the fix was a filter on the 1284 Rx pin. This is indicative of a cross-talk problem partly exacerbated by the long lead frame lines from the silicon chip to the pins on a 40pin package.
The 10K+100picoF filtering in the link works because the Rx pin on the package is right next to the Xtal pins so signals on the Rx data line (which toggles a lot of times during upload) easily couples into the very sensitive xtal oscillator pins and upsets operation. On PCBs a guard ring is typically used in these situations (grounded track that encircles that bit of xtal circuitry) and this redirects the unwanted coupling to GND. The choice of Xtal is also a factor as some oscillate with different tuning capacitance values.
Typically you can buy ready-built 1284 board for less than the cost of the 40pin device, like this one which is $9 and I have used successfully where space is a concern.
This PCB has a good approach where the ground plane protects the Xtal pins from coupling from the Rx line.
