URL Discovery and Web Crawling in SEO

Chapter 7 of SEO Best Practice: Technical Optimisation

Search engines find URLs using web crawlers that inspect a variety of sources. This chapter looks at these sources, what factors affect how search engines crawl a site and how to identify any crawling problems.

By Econsultancy

8 minute read

Search engines deploy web crawlers to discover new URLs. This chapter outlines how SEOs can optimise their sites for web crawlers, covering:

Discovery: The sources web crawlers review and how can SEOs optimise for crawling activity.
Crawl budgets: The amount of time a crawler spends crawling a particular site and the factors that affect this.
Optimisation: A checklist of 10 key steps SEOs can take to manage the crawling of their site.