You must not apply an external voltage higher than Vcc or lower than Gnd to any of the I/O pins without current limiting (aiming to keep current under 1mA maximum). When the board is not powered, Vcc = 0, so any external voltage can cause a problem - the pins are protected with internal diodes to the power rails, but these are not meant to carry much current.
Backpowering the processor through it's protection diodes is an excellent way to blow the pins involved.