If you are not easily scared by higher math here is a starting point
Optical flow - Wikipedia. And of course this has been done http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/, Quadcopter | Hackaday
I once read an article where NASA converted optical mice by means of suitable lenses for greater distance reading. Of course it will never work if you use only once mouse because you can not detect rotation around the sensor axis. But I always wondered (but not yet tried) if it would work using two or three mice. That way it would be possible to detect rotation and compensate for it.