So considering that mouse is 800dots per inch (2,54cm=25,4mm)
by simple equation I got that each step represents movement of 0,03175mm multiplied by 127
equals about 4,03mm maximum in each direction
I was just wondering if sampling rate or resoution affects these calculations
Well, if you move mouse too fast you get max 127 counts in each direction which results in inaccurate measurement due to some not being recorded.
According to wiki ps2 interface, the range should be -255, 255 - it's 9 bit two complement intiger.
However ps2 library uses chars as variables which are 8 bits. Any solution how to add that 9th bit? This should result in doubling maximum distance before going out of range
You could go into the library and change them to int or long. Int goes up to 65,536 + or -. That should work. However, the mouse might output in chars. Try messing with it and post the results.