Sam Crocker, one of our lead SEOs in London has been researching what proportion of searchers
are seeing Google Instant. This has a potentially significant impact on the
various research being published on how Instant affects average query length.
Here’s what he has found:

There has been a great deal written about Google Instant since its release and some very interesting studies about the impact of Instant on search behaviour. These studies have analysed the number of keyphrases of a certain length, the average length of searches, and even gone so far as to point out that the overall impact of Google Instant has been smaller than the impact of the average update to the Google algorithm in terms of search traffic and the length of keywords.

Whilst we can glean some information from this data I want to be very careful to point out that we should not be too hasty. The most important point to make about the referenced studies is: none of these studies claim to have segmented out the data from just Instant traffic. If this is the case, any implications of Instant’s impact on search behaviour does not accurately measure the impact of Instant, but rather the observed change on overall traffic driven by search since Instant has been released – and this differentiation is absolutely essential.

When Google updates its algorithm it impacts everyone but usually will not change user behaviour or experience in any drastic way. However, changing the “Google” experience as significantly as Google has done with Instant- has the potential to change the way we search, rather than SEO performance and this is extremely important to bear in mind.

There is a difference between search behaviour and performance of a website in the search engines. Search behaviour refers specifically to the way people search, whereas performance in the search engines refers more generally to how the search engines rank particular sites for a given word.

Looking simply at “before” and “after” Instant results in analytics does not accurately account for the potential impact of a change in the user experience of this magnitude.

We have taken a stab (given the data available to us) at segmenting the data and analyzing the true impact of Instant on searchers behaviour, rather than the broader impact on search volume and changes in analytics data pre/post the introduction of Google Instant. And, as you may have gathered from above, our results look very different than some of the other studies out there.

Our study is by no means perfect science. However, we are hopeful that it will push Google to share a bit more with Analytics users and that others will run these numbers across their own data but the results we did find would certainly indicate that the prediction of many about the increased importance of the long tail may actually still be valid (see Ian Lurie on Read Write Web).

What’s the Difference?
Well, for starters, Google Instant has created a whole lot of buzz for having a fairly negligible impact on the average user. According to some early estimates by Latitude, Google Instant was only expected to be seen by 2.1% of users.

Meanwhile, in a post by Essential Travel, their team looked at a number of users and ran them through a handful of tests. In their tests- relying specifically on users interacting with Instant- they point out that about 85% of their test subjects typed out the full phrase for which they were searching rather than let Instant work its magic. This study was cause for some debate in our own offices as to whether or not a user hitting enter after conducting a search really counted as using Instant at all.

Whilst neither of these posts really “prove” anything, they are indicative of some of the data we, here at Distilled have looked at in our own log files.

After reading some of these posts (particularly the ones that suggested that Instant has had little impact on the length of search queries) I thought it would be interesting to take a look at the log files for our servers. I admit that this data is by no means statistically significant (it’s a small sample size for one website generally tailored towards people who are active in online marketing and thus are fairly savvy), but it certainly is reflective of the fact that the true impact of Instant is probably being grossly underestimated.

Our Evidence & Our Findings
We examined our logfiles for Google referred traffic to our site from early on the morning of September 19 – the evening of September 24. The size of the sample is too small to lead to anything too conclusive (3,094 natural search visits over 5 days), though it was certainly enough to further cement our suspicion that the studies on Instant have thus far underrepresented its impact on search behaviours.

Perhaps the first thing that jumped out at us is the fact that only 5% of our visits could be traced back to Google Instant with any level of confidence. Although we have yet to find any explicit confirmation our analysis suggests that the addition of “sqi=2” and/or “sclient=psy” and our findings are based upon the referrer query string containing one or both of these elements.

I am happy to concede that this is not a statistically significant study, though it echoes some of our original concerns:

  1. We do not yet have the data to see the true impact of Google Instant
  2. We have been measuring the overall impact rather than the specific impact
  3. We have (likely) underestimated the impact of Instant on search behaviour in a serious way
  4. Very few people appear to be using/seeing Google Instant

Every study we have looked at until now has argued strongly that “Instant doesn’t really change anything” which is true when considering the impact of Instant on all search traffic. However, the caveat to having any significance is that Instant continues to be shown only to a small percentage of searchers (which in fairness has been pointed out in some of the other studies we’ve referenced).

Google Instant may not change much in the way of trends for “all search traffic” though the suggestion that it can tip the scales at all is indicative of the fact that we’re not giving it full credit (have another look at the first graph).

Extrapolating on the data and segmenting by whether the user came to the site by way of Instant we saw:

So what would this mean if everyone were using Instant (i.e. the actual impact of Instant on search behaviour) versus the length of queries of all searches over this period:

  • 2 term keyphrases: 10.3% of searches rather than 14.5%
  • 3 term keyphrases: 18% of searches rather than 22%
  • 4 term keyphrases: 27.5% of searches rather than 23.7%
  • 5 term keyphrases: 12.9% of searches rather than 16%
  • 6 term keyphrases: 10.3% of searches rather than 15.1%
  • 7 term keyphrases: 21% of all searches vs. 8% (almost 3x as many visits)

This lends some traction to the idea that the true impact of Google Instant on searchers behaviour has likely been grossly underestimated. The net effect (at the moment) may be rather small, but the potential impact should Instant ever become the norm is massive. And suggestions that Google Instant has not had any noticeable impact on long-tail search should be challenged as the product is rolled out to a larger audience.

Perhaps the above chart is the most helpful in illustrating the potential impact of Google Instant as compared to the current state (e.g. some users with Google Instant, some coming by way of the toolbar and some by way of “traditional” Google) or the “Percentage all natural” case where no search would be coming by way of Instant.

The comparison of the Red and Blue data points suggest that perhaps there has not been a very big change. This is consistent with these other studies’ findings in large part because it discounts for the fact that such a small minority is actually using Google Instant.

The case in green in the above chart is unlikely to ever occur (if for no other reason than the fact that people may always prefer use the toolbar for search and so on), however it is important to illustrate the problems with the conclusion that Instant has not changed anything. The indications from this (admittedly too small) dataset are that Google Instant may yet be a “game changer” in the search marketing community.

What should I do!?

The answer is not to panic, and to keep your eye on this. At the moment, it would appear as though Instant is only impacting a limited number of searchers. We wouldn’t want you to change your approach drastically as a result of this research but we do want to challenge the mantra that “Google Instant hasn’t changed anything”.

We should probably be paying more attention to the uptake of this new product (5% is a shockingly small percentage) much more so than its implications on analytics when so few people are using Instant at present… and we should definitely keep testing. So far the impact may have been fairly negligible but the value of the longtail could very well increase if rollout does.

Google Instant almost certainly has changed search behaviour for those using it- the ultimate unknown is just how many people this is ultimately going to impact and how many people end up using Instant search.