If you’re a regular reader of SEO websites, or have read any generalised SEO best practice guides, no doubt you’ll have come across many references to the importance of technical SEO.
Technical SEO, which refers to the elements of search optimisation that help search engines crawl and index your website, is often described as the “foundation” of search optimisation. It can also be – as the name implies – technical, and often technical SEO techniques will overlap as much with the work of a developer as they do with the work of a marketer.
However, that doesn’t mean that you need to be a technical specialist to carry out a technical SEO audit of your website – even if you’re a complete beginner. At last week’s Brighton SEO conference, Helen Pollitt, Head of SEO at Reflect Digital, presented a highly accessible introduction to technical SEO for beginners, packed with tips to get you started on your first SEO audit.
Here are some of her tips.
1. Checking your website
One big obstacle to spotting potential technical SEO issues is that most of the time, you’ll be accessing your website using the same device and the same browser – often a desktop computer, with a widely-used browser like Chrome or Safari.
But people will be accessing your website from all manner of different devices and browsers, and ideally, you’d like each of them to get the same great experience. So, the first tip that Pollitt gave was to “dust off Internet Explorer” – yes, really – and view your website in a browser that is still used by 3-7% of web users. How well does – or doesn’t – it perform?
Then, of course, there’s mobile. We are firmly in an age where mobile use unquestionably outstrips desktop, and Pollitt stressed that the version of your website that appears on mobile should be the best version of your site.
You should also avoid any uses of Flash like the plague: HTML5 will give you all of the same functionality, without any of the resulting issues with mobile.
2. Crawling your website
Search engines use crawlers, also called search spiders, to index the contents of a website. However, crawlers are also tools that you can use to find out what’s going on “under the hood” of your website.
Screaming Frog is a great tool for this and probably the most widely-recommended by SEOs. Its SEO spider is also free, for up to 500 URLs.
One handy thing you can do to check for technical SEO issues on mobile is to set your crawl bot to mimic Googlebot for Smartphone – though Pollitt warned that some websites may unfortunately block bots that are imitating Google, in which case you’ll need to fall back on a regular crawler.
Other things to look out for when you crawl your website:
- Make sure you have set your bot to crawl sub-domains, like m.domain.com
- Assuming that your site uses HTTPS – which it should! – check for any HTTP resources that turn up during a crawl, which indicate that your site isn’t as secure as you thought it was
- Check that none of your pages are returning anything other than a 200 status code (which is an “OK” status) – and especially that there aren’t any 404s
- Check that your directives – noindex, nofollow, canonical tags, rel=”next” and rel=”prev” – are implemented correctly. By default, Screaming Frog will not be configured to crawl internal or external “nofollow” links or rel=”next”/rel=”prev” elements, so make sure you check these options when configuring the crawler
- Look out for orphans (pages with no internal links pointing back to them) and low-linked pages – these are effectively you telling Google that this page is unimportant, since you don’t consider it worth directing visitors to! Your crucial pages should have plenty of internal links pointing to them
- Look out for “spider traps” – pages that might have been automatically created without you intending to. If you have a relatively small website that takes a very long time to crawl, this is a tell-tale sign that there might be pages you didn’t know about on your site.
Configuring Screaming Frog SEO Spider to crawl all subdomains.
3. Using Google Search Console
Google Search Console is like a “dashboard” for webmasters and SEOs to check up on the status of their site in search. You’ll need to sign up in order to use it (it’s free!) and add your website to your Search Console, which involves verifying you own the site.
There are various ways to do this, including uploading an HTML file to the site, adding an HTML meta tag, or adding a Google Analytics tracking code or Google Tag Manager container snippet. Depending on your level of access and the tools you use, you might need a developer to carry out the verification.
The Notifications section of Google Search Console flags up developments with your site that you need to be aware of, such as index coverage issues it encounters while crawling your site, and manual actions (hopefully you won’t get any of these!).
Even if you’ve already used a crawler to check for issues on your site, Pollitt noted that the big advantage of using Google Search Console is that Googlebot can find pages that are linked elsewhere online that you might not know existed – whereas regular crawlers are confined to what’s linked to on your site.
You should also check your disavow file (a list of links you’ve asked Google not to take into account when assessing your site, e.g. spammy links) for any important, genuine links that might have been disavowed by accident, possibly because they seemed suspect at the time.
Disavowing links in Google Search Console
4. Submitting your XML sitemap
An XML sitemap is a file saved in the format .xml which lays out the structure of your website and helps search engines to crawl it. It can help indicate to search engines which pages on your site are important and how often they’re updated, as well as tell search engines about pages that exist which might not have been indexed.
Screaming Frog’s SEO Spider tool can generate an XML sitemap for you after it has finished crawling your site. Here is a step-by-step guide to using the SEO Spider to generate a sitemap.
If you have an existing XML sitemap – which can also be created via a plugin on a CMS like WordPress, or built manually – you can also use Screaming Frog to audit it for errors by uploading it in list mode (BuiltVisible has a step-by-step guide here). Check that there are no server codes displaying except 200, or pages missing from your sitemap.
Uploading a sitemap in list mode on Screaming Frog
Finally, upload your XML sitemap to Google Search Console to help Google’s crawlers understand your site. The Sitemaps Report in Search Console allows you to view and submit sitemaps, and check for processing errors.
5. Using Robots.TXT
Robots.TXT, explained Pollitt, is a file on your website “for communicating with robots”. Not our future robot overlords, mind you, but web robots – most notably search engine crawlers.
Robots.txt files are used for indicating whether web robots can and cannot crawl certain parts of a website. The Robots.txt file for your site can be found at [yourdomain]/robots.txt.
Be aware also that if you’ve blocked a page using robots.txt, search spiders won’t know whether you’ve added a noindex meta tag to the page’s HTML (to prevent it from being indexed), and it might still appear in search results, for example if another website links to it.
6. Speeding up your site
Site speed is extremely important for a good user experience, and it is also a Google ranking factor on both desktop and mobile search (though at present, Google’s mobile search only penalises the very slowest websites).
Pollitt recommended the Pingdom Website Speed Test for checking desktop speed, and Google’s mobile speed checker for mobile.
The server is often the worst culprit for slowing down websites, which is something that you would need to talk to a developer or your domain provider about in order to sort.
Other site speed fixes include storing some of your code in Google Tag Manager, or using a Content Delivery Network, which is a geographically distributed group of servers working together to provide fast delivery of internet content. It isn’t a substitute for web hosting, but can help to relieve some of the pain points of traditional web hosting, for example by using caching to reduce bandwidth.
To take your technical SEO auditing to the next level, download Econsultancy’s SEO Best Practice Guide to Technical SEO.
Econsultancy also offers training in SEO.
Really liked this article. Would love to see if there is a video version for Helen Pollitt’s presentation.
I would like to add the benefit of using “site:” command to check the indexed pages of website during technical seo audits.
It usually reveals some interesting indexed urls that might point to a issue that might have missed otherwise.
I am doing seo for some time now (currently working as seo analyst in Global Media Insight, an seo agency based in Dubai), and find this quiet useful while performing technical seo audits.
I have done one of this searches during my recent audit and found that there are close to 2500 indexed pages for this website when I know that the total pages in website is definitely less than 500. This pointed out to a content duplication issue which I fixed later.
I had de-indexed the duplicate urls and redirected/blocked them which reduced the count of indexed pages.