Can you use two IR transistors and and IR LED to pan a turret?

What I'm asking about is essentially the same thing as a sun tracker.

except it will need to follow an IR LED or small cluster of LEDs and be much faster on the response time. I know the servo can move much faster but maybe there is something else limiting the speed like the response time of the phototransistor. Do any of you with experience with this stuff think that it will be able to work with those modifications?