In order to diagnose server configuration problems that could affect a search engines’ ability to crawl a website, SEOs must understand how web browsers and web crawlers communicate with web servers. This chapter covers:
- Information exchange: Information is returned in request and response headers, which can be viewed through the search engines’ tools, third-party tools or browsers.
- Response status codes: The most important piece of information returned in a response header is the three-digit status code. What are the most common response codes, and what do they mean?
- Tags: Various tags can also be included in the headers to tell crawlers more information about the page. What tags might SEOs consider adding to the header?
- Detecting web crawlers: Businesses may need to detect and identify crawling activity from specific web crawlers, for example to check for competitor activity incurring significant server costs. What steps can SEOs take to detect web crawlers?