If your Pot is connected as in your diagram I would expect it to produce ADC values from 0 to 1023 - but maybe the joystick levers cannot move the Pot through its full range.
If the range is from 319 to 567 it would probably simplify the maths if you subtract 319 so that the range is from 0 to 248.
And if you subtract a further 124 (making 443) you will get a symmetrical range from -124 to + 124.
And 124 * 2 is near enough to 255 to make little difference.
...R