With original setting the crystal oscillator of the 1284p is set to a low-power mode. The oscillations are weaker and the oscillator's input (XTAL1 pin) is much more prone to be pulled/modulated by an external signal. As the analysis has shown it may even come to a stoppage of the oscillation.
The closest pin to the XTAL1 is the RX0 (uart input). When the data come to this RX0 pin from FTDI or other serial data source, the fast leading/falling edges of the serial data may pass through a parasitic capacitance, call it Cpar (Cpar is the capacitance between the XTAL1 and RX0 pins) to the oscillator's input (XTAL1) which is very sensitive in low power mode. The faster/steeper the edges of the data signal the easier they pass through. Even a small Cpar (a few pF) may let pass the serial data signal to XTAL1 pin and thus influence the crystal oscillator frequency (even shut him down).
When you put a serial resistor into the serial data path, you introduce a low-pass filter (RC), thus the edges of the data signal are more rounded, not so steep (not so fast, not so energetic, with less high frequency content), and they cannot pass via Cpar easily (because the Cpar is small), so the negative effect on the oscillator is smaller. Therefore it helps.
When you set the mode of the crystal oscillator to "Full Swing Oscillator" the operational condition of the oscillator stage changes, the oscillator takes more power (it becomes a harder signal source), its amplitude is much larger (rail to rail swing, saturated), the input of the oscillator (XTAL1) less sensitive, thus the oscillator is much less prone to be influenced by a parasitic feed through of the energetic edges from the serial data to the pin XTAL1.
The actual issue depends on other factors as well - ie.:
. PCB design (how much Cpar you create, do you use GND guard ring around XTAL1 pin??, are the traces leading to the XTAL1 pin short??, etc),
. crystal parameters (ie some crystals are easier to be driven into oscillation, other not..),
. the oscillator design itself (the circuit inside the chip, may vary based on the chip version),
. frequency of the crystal,
. voltage (the lower the voltage, the lower the recommended crystal frequencies),
. temperature (it shifts the crystal oscillator operational conditions),
. decoupling (bad or missing decoupling capacitors mean a lot of mess on the power lines - that mess modulates the crystal oscillator as well, especially in the low power modes or at lower voltages).
That is my current understanding..