Yahoo! recently announced the launch of a new faster, more efficient web crawler. The new crawler (
as it’s known), is designed to navigate through the Web quicker than its predecessor and in the process uses 25% less bandwidth.
You might have seen far more activity from the Yahoo! Slurp crawler in your website logs as Yahoo! has been running the old and new crawlers concurrently over the past few weeks. The extensive testing is designed to ensure that the new crawler lives up to its expectations.
The launch of the new crawler was announced by David Simpson on the Yahoo! Search Blog. David comments:
“In addition to crawling the Internet faster, our new crawler is more efficient at visiting websites. As a result, site owners will notice as much as a 25% reduction in the number of requests and bandwidth consumed by the crawler.
Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.”
This is good news for website owners, as each time the crawler visits your website it consumes your bandwidth when indexing your web pages. For owners with small bandwidth hosting packages, these savings will be appreciated.
Yahoo! has also suggested that the new crawler will inevitably lead to some changes to search engine rankings and reshuffling of web pages within their SERPS.
Initial reviews about the new Slurp have been mixed. While some website owners are happy with the drop in crawl load to their sites, others are not overly impressed with the new search results.
So why not check out the new Yahoo! search results and share your thoughts on Yahoo!’s new crawler? How has Slurp affected your rankings?