In a bore of only 3 mm the thickness of the caliper blades will necessarily cause a significant underestimate. I suspect that the bearing is the right size for a press fit on your shaft. As others have said, you can use temperature to help, but no matter how you do it pressure should be applied ONLY to the inner race, not to the balls. I generally use a wrench socket with a loose fit on the shaft and a vice, or for longer shafts, a bar clamp.
Ciao,
Lenny