Google’s recent introduction of SSL encryption for search queries from logged in users means that a lot of valuable data has now gone missing from Google Analytics.

This post explains a useful Google Analytics hack to regain some of the insight we used to have, and improve life for you in light of the ‘(not provided)’ issue.

First, here are three devastatingly important ‘notes of caution’:

  1. You need Google Analytics Admin access to apply the ‘hack’ included in this post.
  2. If you get the hack wrong, it can mess up your data a little bit while the hack is in place.
  3. Econsultancy & Dan Barker take no responsibility for your implementation of this little hack. (though feel free to get in touch with Dan on Twitter if you want to ask questions).

Background

If you haven’t been keeping pace lately (and, with the changing of the clocks, and an utter dearth of new David Duchovny films, who could blame you?) here is the background on this:

Before October 18th 2011, whenever visits landed on your site from its search results pages, Google would happily tell you all of the keywords they’d searched for:

After October 18th, Google started hiding(!) lots of this information:

(CC images courtesy of jurvetson & marriageequality on Flickr)

In other words: Before, for every 100 visits from Google, very roughly speaking, you’d see what all 100 searched for; now, between 5% and 33% of those terms are hidden by Google.

You still get 100% of this info for ‘paid search’, but for ‘organic search’, much is hidden. How mean-spirited of them, eh? The worst case I’ve seen of this so far is a site with more than 10,000 ‘(not provided)’ search terms each week.

For further background, an original announcement of this is here. & there’s much discussion & opinion on this here and here, among many others.

How to fix this?

The bad news is you can’t completely fix this. Sadly, the scallywags at Google have stuffed this data right into the back of the larder, though we keep our fingers crossed. 

The good news is, there are some things you can do:

  1. Firstly, the great Avinash Kaushik has put together a magical blog post detailing how to benchmark the effects of this, and to run some other, new reports to assist.
  2. Secondly, I’ve put together a nice little Google Analytics hack below to bring some useful data back into your regular Google Analytics organic ‘keywords’ report wherever ‘(not provided)’ would now appear.

The hack: what it does

I’d said the hack puts ‘some useful data back’ into your keyword reports. What it actually does is this:

  1. Looks for ‘(not provided)’ search terms.
  2. Where it finds them, it looks to see which page the visitor landed on.
  3. It then changes your keywords report in Google Analytics to show those two pieces of information (the fact that Google suppressed the keyword, and the landing page), rather than just the utterly anonymous ‘(not provided)’.

The hack: why it’s useful

The hack will give you the following insight that ‘(not provided)’ hides:

  • Get a much clearer picture of the intent of all of your (not provided) visits.
  • Get a vague idea of the search terms those (not provided) visitors may have used.
  • Quickly get a rough idea of the ‘brand’ vs ‘non-brand’ split, by simply seeing which (not provided) visits land on the homepage vs others.

I had originally been running additional reports to do some of the above, but I’ve found it far quicker & more useful to add that data back into keywords reports so I can analyse ‘(not provided)’ data alongside my regular search data.

The hack: before & after

Here’s a very quick example of the hack on a small amount of data.

The Keyword Report

The Google Analytics Keyword/Organic Report looks like this:

Not provided – before the hack

If we ignore everything else, and concentrate purely on the ‘(not provided)’ section. Here’s the information Google now gives us on these anonymous keywords:

Obviously, there’s not a whole lot of insight we can gather from that. Are those brand terms? Are they generic? Are they long/short tail? There is no way of knowing.

Not provided – after the hack

Here’s the information we now get about those ‘(not provided)’ keywords:

Even from that small example, you can quickly see things like:

  • The vast bulk of ‘not provided’ data was from people landing on the homepage. (ie. a bulk will probably be ‘head’ terms).
  • ‘Laptops’ & ‘cheap laptops’ are next biggest landing pages among ‘not provided’ terms. I know these are likely to be driven very much by ‘head’ terms.
  • The rest is a little further down the tail of search.
  • The ‘bounce’ rate varies wildly among this, as opposed to the flat number we saw before.
  • We could also use this to see any e-commerce or goal data related to (not provided).

And that’s purely an example on a tiny amount of data. On larger pools, there’s a whole lot more insight you can gather.

The Hack: how to apply it

The hack uses a ‘filter’ to change the data Google Analytics records. If you’re at all uncomfortable about this, you may want to test it on a ‘duplicate’ profile first.

To apply the hack, first we have to go to the ‘filters’ area.

Step 1: Add a new Filter

Here’s a diagram showing how to reach the filter admin area in the ‘New Version’ of Google Analytics:

Step 2: Configure the Filter

To do this, we create a ‘Custom Advanced Filter’ (this sounds more scary than it is). Here are the settings to apply:

This tells Google Analytics:

  1. “Whenever you see a search term that matches ‘not provided‘, apply this filter”.
  2. “Within this filter, we also want to use the address of the page reached where ‘(not provided)’ was the referring search term”.
  3. “When you apply this filter, discard the original ‘(not provided)’ search term, and replace it with ‘np -‘ followed by the landing page address”.

Clicking ‘Save’ then applies that filter to the Google Analytics profile, and starts recording the data. (note: you can edit and remove filters at any time later).

Step 3: Test the Filter

To test the filter:

  1. Go to https://www.google.com (if it redirects you to Google.co.uk or another international site, you’ll need to click the ‘Go to Google.com’ link at the bottom right of the page).
  2. Make sure your browser address bar contains ‘https://’.
  3. Search for something you know you rank for. Preferably a page deep within your site.
  4. Click the link through from Google to your site.

If all has worked, after a couple of hours you should be able to view the keywords/organic report in Google Analytics and will see that you have an entry beginning ‘np -‘ followed by the page you landed on.

Following that, you’ll start to see reports like this:

Instead of just this!

Summary

So there we are. Not as good as the data we had before Google took it away, but far more valuable than the single ‘(not provided)’ line we get as standard at the moment.

There are lots of possible extensions of this too. For example, you could clean up the URLs you see here if you have lengthy ones, use ‘Page Titles’ instead of URLs if more useful, etc.

Let us know what you think in the comments, and do share this via Twitter if you think others may find it useful.