In a long thread on Stack Exchange about why the site’s speed had seen an uptick recently, co-founder Jeff Atwood dropped this interesting nugget.
Google’s search spiders crawl SE’s to index its wealth of answers at a rate of ten times per second, which is the maximum the Mountainview giant will throw at anybody. According to Atwood, this was gumming up the works a bit.
“Google doesn’t need all the same work done per page that, say, a user does — and when Google hits thousands of pages in a few minutes, that can kick off a lot of background work, such as rebuilding related questions. Not expensive by itself, but when multiplied by a hundred at once.. can be quite painful.”
He also popped in a chart which shows the average number of requests from Google has nearly doubled since this past December.
It’s a great sign for SE’s business, because it shows that Google considers them a potent source of constantly evolving information. Just as the search giant recently tweaked its algorithm to try and remove spammy content farm links from its results, it has apparently upped the emphasis on the kind of in depth, long tail question and answer material contained on Stack Exchange.
(h/t PeGobry)
