One way of removing it is to use an operational amplifier instead of a transistor. You wire it as a voltage follower, that is the wiper to the + input and the - input connected to the output. You need to use a rail to rail op amp to maximise the voltage range. Also op amps do not provide much output current (see the data sheet of the one you choose) but they can be in the region of 50 - 100 mA.
The different LED colours, and hence the cut off point voltage was always a problem with the concept of controlling the brightness of an LED by controlling the voltage to it, and hence the use of a digital pot to dim an LED is not the brightest of concepts. Which is why PWM is nearly always used to control the brightness od an LED.
As a total fudge you could arrange each different op amp to provide a suitable DC offset on the two colours to "match them up" so they drop below the LED's turn on voltage for the same wiper voltage. But why bother?