What have you tried?
Have you got a program that can move the motor back and forth?
Have you got a program that can count the encoder pulses and display them on the Arduino Serial Monitor?
If not, get those working separately first.
One huge difference between and encoder and the potentiometer in a servo is that the encoder does not know where it is. (Unless you are using an expensive absolute-position encoder). That means you will something (such as a microswitch) that is triggered when the shaft is at a known position so that the Arduino can use that as the zero-point for counting pulses.
You will need a h-bridge of suitable capacity for your motor.
Is the output shaft geared down from the motor shaft? Servos have a huge gear reduction between the motor and the output shaft which means that small movements of the motor as it keeps position have very little impact on the output shaft. Also the gearbox provides a lot of friction so that the load on the output shaft has to be high before it can move the motor.
If you need more information please post a link to the motor and encoder datasheet.
...R