I wonder why it takes Microsoft 20 bots to do the same job Google does with 1?
Microsoft is a small fish in a big pond whereas Google is a whale. Microsoft has nowhere near the server power that Google has.
From what I can tell, the vast majority of the time, Microsoft uses its own tools. Occassionally, there are reports that Microsoft executives have come down hard on employees who use something that wasn’t built by Microsoft.
I suspect Microsoft is using the closest fitting Microsoft tool that works.
Google, on the other hand, is very likely using the best possible tool for the job.
Because it’s Microsoft ;D
Remember, these are the people who gave up on writing an OS for desktop systems and switched to requiring everyone to buy server-class machines.
a. had difficulty distributing the webcrawling results among their various server complexes, and wound up having each cluster run its own bots, or
b. don’t give a (bleep) about the costs they’re dumping on the webserver owners, and are gathering data in a way most convenient for themselves.