I do not see any drawing or other information on the machine or how the encoder knows the movement of the metal sheet. I suspect it is ALL friction driven and the friction is NOT consistent through all the operation.
Be aware of using serial comms too, that can add unwanted delays although you’ve got nothing but Serial.begin(9600); so it may not be an issue. With the Mega2560 you can use serial 1, 2 and 3 for external comms and still use the D0 and D1 pins for digital IO.
Serial1.begin or Serial2.begin are the way to use the extra serial ports.
9600 baud is extremely slow too, use at least 57600 unless there’s a special need.
Just comment out the Serial.begin as a test.
Hi, @gvidmar
These encoders do not contact bounce.
Did you write some simple code to check your encoder output, or did you use an example from the Library?
Tom.... 
I didnt use a library..i dont know why it works to be honest..it just does 
i just added the code i wrote above. Also, i forgot to mention that at first i had forgotten to delete the library part in the code as i wrote it on my own so there was probably an issue there too, but it didnt work untill i added the last piece of the code. Also @xfacta the serial.begin was only for when i tested the encoder in the serial monitor.
I am aware i was lucky here..but still..if it works it works
Anyone here who alredy made a similar project?
Yes, if it ain't broke don't fix it.