the fan is still pretty omnidirectional.
That's not necessarily a bad thing: the low (not really "non-") directionality means that the pressure decreases as the ball gets farther from the fan. It's not an inverse square, like a truly omnidirectional source, but it should decrease according to some mathematical function.
With the ball in the only-slightly-larger tube, the pressure is relatively constant until the ball reaches the top, and the leaking gets significant.
It might help to drill holes in the tube, so the higher the ball goes, the more the air leaks out.
Or it might help to set your nominal "zero" around the top of the tube. You should have a more- (but not perfectly-) directional flow coming out of the tube than out of a "naked" fan, so the change in height of the ball will be closer to a linear function of the change in fan speed.
I know this sort of thing has been done before. Some google searching might turn up some reports on their experience.