ReadWriteWeb

Technorati Retiring Old Crawler

Written by Lidija Davis / February 7, 2009 1:55 PM / 2 Comments

technorati_jan_09.jpgAccording to a post on the Technorati blog today, the original blog search engine is in the process of replacing its old crawler with a new design in an attempt to track and index the blogosphere more efficiently.

While the company has been working on this new infrastructure for over six months and the larger blog platforms have been using the new crawler for some time, Technorati expects to see all blogs utilizing the new crawler by the end of next week.

"Over the last six months we've been working on a rewrite of our blog crawling infrastructure from the ground up." Dorion Carroll writes in the post. "With a new architectural approach, we've added a lot better coverage of many new blogging conventions, better visibility and tracking of what is happening in the internals of the crawler, and how to do a better job of recognizing all the great content bloggers are discussing and linking to."

Over the past year Technorati has been getting a lot of attention for its inability to crawl and index blogs efficiently, so this may be great news for bloggers who have for a long time bemoaned having to manually ping Technorati each time they publish a new post.

And it's not only the smaller blogs that have experienced this. Only yesterday, while looking at the top 100 blogs in Technorati, we noticed that ReadWriteWeb had not been crawled in 23 hours. That's a long time considering we had published ten posts and therefore pinged the service ten times in that period of time.

While we can't be certain that this new crawler will make indexing faster, we're pleased to see Technorati striving to become more relevant and useful at a time when really great, really fast blog search is essential.


Comments

Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts

  1. Let's hope for the much needed improvement.

    Posted by: Engago Team | February 9, 2009 12:43 AM



  2. so this may be great news for bloggers who have for a long time bemoaned having to manually ping Technorati each time they publish a new post.

    Posted by: mirc | June 14, 2009 10:03 AM



Leave a comment

Optional: Sign in with Connect Facebook   Sign in with Twitter Twitter   Sign in with OpenID OpenID  |  
RWW SPONSORS


FOLLOW @RWW ON TWITTER

ReadWriteWeb on Facebook



TEXT LINK ADS