Infinite Scroll and SEO

What is it? 

Most of us engage with infinite scrolling on a daily basis, most frequently with Facebook, Twitter and Pinterest.

It occurs when you scroll down to the bottom of a page of tweets, posts or other content and the page automatically updates to append more to the bottom, and does so again and again as you hit the bottom of the page each time (the scroll isn’t infinite however as the name misleadingly suggests).

A variation of infinite scroll is infinite ‘page’. Whilst the underlying architecture is broadly the same, the intention of each is quite different.

With an infinite scroll the expectation is that more and more content is added to the stream at regular intervals, whereas with an infinite page the expectation is that the narrative has a clear beginning and end and it won’t really be added to if at all (so even less infinite than the very finite infinite scroll).

A good example of an infinite page is the Renault Ze website which uses infinite scroll to tell a story and is not designed to regularly accommodate new content.

 

What are the SEO impacts? 

Search engine spiders cannot scroll to the bottom of a page like an ordinary user, trigger the request for more content, and then wait to retrieve it for indexation.

Therefore, a web-page utilising infinite scroll will by default serve the search engine spider the first page of results and nothing more.

This means that lots of great content may never be seen by a search engine if there is no other mechanism in place to serve as a path to that content.

Content only accessible via the infinite scroll will simply not be seen and therefore will not make it into the respective search engine results listings.

You can check whether you have an infinite scroll or infinite page issue by emulating a search engine’s experience with your site.

You can do this by checking what Google has captured of your infinite scrolling page – just type in ‘cache:www.yourdomain.com’ into Google and click on ‘Text Only’ and see what is missing and what the crawl path to your content looks like.

Alternatively, you can (and should) also use a Web Developer extension on your browser that allows you to turn JavaScript off with just a few clicks and experience the website in that impaired condition.

What are the possible solutions?

Various sources suggest that infinite scroll should be avoided as it is bad for SEO. That is incorrect and unfair – anything poorly designed and executed is likely to be bad for SEO and infinite scroll isn’t inherently bad.

There are easy solutions so that infinite scroll poses no problem for SEO and in fact it can also provide an opportunity to maximise user experience and maximise SEO value at the same time.

The most popular and most effective solution is to implement some graceful degradation, otherwise known as progressive enhancement. This is where web techniques are used in a layered fashion so that every user receives a comparably functional and pleasant experience regardless of the capabilities of their browser or device.

So for basic browsers, a site with progressive enhancement will gracefully degrade the user experience without impacting accessibility negatively, and an advanced browser will find their user experience elevated if it meets various capability criteria.

A web page that offers infinite scrolling therefore should degrade when utilised by a browser or device with limited functionality, particularly one that does not, or has an inconsistent approach to executing JavaScript.

Search engine spiders would qualify in that browser/device category.

Therefore, when the page is viewed by a search engine spider the site should automatically provide a suitable replacement for the infinite scroll.

For most sites, this replacement is the appending of traditional pagination links, e.g. Previous 1 | 2 |3 | 4| 5 | 6 | 7 | 8 | 9 | 10 | Next.

This is ultimately achieved by either placing the pagination within a noscript container in your page code or layering JavaScript, CSS, and mark-up independently of each other in your code so that the pagination is simply obfuscated progressively as a users’ device and browser allows for it (i.e. the pagination still resides on the page, even if you can’t see it when you’re infinitely scrolling in a contemporary browser).

This second approach would be considered the most desirable, mainly due to concerns over the relative value search engines place on content within a noscript container given it isn’t designed to be universally seen by all, and a general suspicion of the container due to its aggressive utilisation by spammers historically. 

Google’s rel next/prev tags can also be used with this pagination to control how many of your first few pages you’d like Google to treat as being at the same level hierarchically.

This is useful when you’re not adding new content all the time and/or when the topic of your items (posts for instance) are highly searched for within search engines and will continue to be so for a while, so Google viewing them as residing on your homepage (or higher up than they actually reside) for longer, may be very desirable. 

Some implementations take this progressive enhancement a step further and replace the infinite scroll functionality with an indexable archive, typically broken down by months of the current year, and then previous years, leading into a further breakdown into months on subsequent pages.

Mashable’s infinite scroll is a good example. It degrades for search engines to the most recent content, and then ends with this:

This achieves two complementary goals. Firstly, it means that a search engine can get to content that is below the infinite scroll trigger point. Secondly, this also acts as a great archive ensuring that there is a crawl path to every single piece of content ever produced by Mashable.

Logic can also be built into this archival mechanism to ensure that the crawl path (the number of clicks away from the homepage particularly) is reduced for the most popular pages (based on historical visits or shares, for instance). 

Purists may be concerned with Mashable’s implementation as the site is ultimately showing the search engine different content than you are showing to most users.

This does not however qualify as cloaking as the intent is clearly not to game the search engines but rather to facilitate their traversal through the site because they cannot behave like a human being and keep scrolling until they reach the very bottom.

There is a line to be drawn here though, as the solution would also allow you to insert a footer into your pages for those people (and search engines) that cannot physically execute the infinite scroll.

Intent becomes very important at this stage; a footer that includes links to an ‘about us’ page, a sitemap and a copyright notice is clearly not designed to game any one network.

Conversely, if the obfuscated footer region includes lots of commercial keywords and looks as though it is designed just for search engine ranking purposes, you do run the risk of raising the suspicions of Google et al and incurring a penalty as a result.

Best practice with footers, within the infinite scroll and degradation context, would be to either create a sticky footer (see an example here), which you need to ensure is accessible to search engines itself.

Another option is to relocate the footer to a sidebar visible to everyone – see any Twitter page (Coca Cola’s here) for a great example of this, or any personal Facebook newsfeed (which are uncrawlable of course by search engines).

This second approach would be considered optimal as it is difficult to implement a sticky footer that users actually like and appreciate, and that ultimately adds value to both you and them.

Facebook incidentally is also a good example of how not to do this as Facebook brand pages have a footer that appears at the bottom of the page momentarily when you visit the page, and then disappears forever when the infinite scroll script kicks in. 

Ultimately, infinite scroll has proved a valuable UI innovation for many brands and for many it has become a defining element of their publishing mechanism.

The hard part is determining whether your users want infinite scroll, whether it improves their engagement levels, and whether it is too much effort for too little return.

Making infinite scroll work from an SEO perspective is in fact the easy bit, assuming you select the optimal solution and implement it properly (see solutions grid below).

Pros/cons solutions grid