Thank you everyone!
I had the same problem as amaggio, while using 5V. I also reduced my sketch and board to the bare minimum of the crystal and ftdi and at about every 3rd to 5th Serial input the Atmega1284p resettet itself!
My workaround was to use the pins of Serial1, which didnt show this problem.
But setting the Low fuse to 0xf7 solved my problem.
Thanks!