Thank you all for your fast replys and great advise. I now have it running alot smother and more fine control 
retrolefty:
The quickest thing you can try is to change the interrupt mode, that will change the step speed in half. You will have to see if that is improvement enough or not.
I Did this and it did help but found it was still little fast.
AWOL:
What you could do is divide the incoming pulses, simply by counting them, and only outputting a pulse every three or four interrupts.
Don't forget to reset the counter.
I did this aswell and found combination of the too did the trick nicely.
im not sure if how I went about doing it in my interrupt code is best method. any feedback would be great
//************************************Rotary Encoder**************************************
void doEncoder(){
if (digitalRead(encoder0PinA) == HIGH) { // found a low-to-high on channel A
if (digitalRead(encoder0PinB) == LOW) { // check channel B to see which way
// encoder is turning
Rotate = Rotate; // CCW
}//end if (digitalRead(encoder0PinB) == LOW)
else {
Rotate = Rotate; // CW
}//end else
}//end if (digitalRead(encoder0PinA) == HIGH)
else // found a high-to-low on channel A
{
if (digitalRead(encoder0PinB) == LOW) { // check channel B to see which way encoder is turning
Count ++;
if (Count >= 5) {
Rotate = Rotate +jump; // CW
Count =0;
}//end Count if
}//end if
else {
Count ++;
if (Count >= 5) {
Rotate = Rotate -jump; // CCW
Count =0;
}//end Count if
}//end else
}//end else if (digitalRead(encoder0PinB) == LOW
}//end subroutine
Moderator edit: corrected tags (hopefully)