Setting non-linear steps from rotary encoder for motor speed control

    //check speed every 50ms rather than on each loop
    cli(); //stop interrupts happening before calculation
    if((millis() - encoder1.time_old) >= speed_check_period) {

Why are you stopping and restarting interrupt handling on EVERY pass through loop()? You should do that ONLY when it is time to copy the variables used in the ISR and ONLY for long enough to copy the values.

Before you can map encoder speed to an amount to increment, you need to define what that relationship is. Draw a graph, post a table, something to show how much you want to increment by when the speed is x.