Your original figures give you quite a bit of precision (1 in 152). Is that not enough?
Consider that this is a forward backward control with center off, so in fact I have then only 0-76 each way.
Then, I set some dead band around the center, reducing it even more.
I also have a quick calibrate function for the maximum ends where I store the center point and maximums in EEPROM, this must be within the range.
There must also be some leeway for the center point calibration.
It is getting a bit skinny now

Only then do I
int speed = map(potValue, centre+deadBand, maxTravel, minSpeed, maxSpeed)
for forward speed and Speed is within 0-255 for the PWM