okay I found the problem myself. When I connected the arduino to an external powersource it began to misbehave. when it was powered by the usb itself but still connected to the rest of the circuits ground it worked as intended. Well at least with a little test I have just done on a prototyping board. I have still to try it when it's connected to the encoder. I'll write how that went tomorrow.