{{ searchResult.published_at | date:'d MMMM yyyy' }}

Loading ...
Loading ...

Enter a search term such as “mobile analytics” or browse our content using the filters above.

No_results

That’s not only a poor Scrabble score but we also couldn’t find any results matching “”.
Check your spelling or try broadening your search.

Logo_distressed

Sorry about this, there is a problem with our search at the moment.
Please try again later.

Like so many others, you've decided to revisit your business model and paid content looks awfully good at the moment. Running an online subscription service can be very rewarding, but it's tough.

One of the challenges posed by a paywall is the paywall's impact on SEO. Since content is restricted to subscribers, Google can't spider your content. What can you do about this?

Disclaimer: the technique described here is cloaking and when it comes to cloaking vis-à-vis Google, you should educate yourself on the subject before deciding whether or not it's worth implementing. The purpose of this tutorial is not to advocate cloaking, but to explain how it can be used with paid content since paywalls are (once again) becoming popular with website owners and they can have a significant (negative) impact on SEO.

While there are plenty of sites, some high-profile, that employ this technique without penalty from Google, others have, over the years, reported being dropped completely from Google for what's described here so you'll want to think long and hard about this and weigh the risks against the potential benefits. Caveat emptor.

The PHP Function

Detecting Googlebot's visits to your website is quite simple. Here's the PHP function I use:

function is_google() {

    if( eregi("Googlebot", $_SERVER['HTTP_USER_AGENT']) ) {
        $hostname = gethostbyaddr($_SERVER['REMOTE_ADDR']);
        if( eregi("googlebot.com", $hostname) )
            return true;
        else
            return false;
       
    } else {
        return false;

    }

}

What it does:

1. Checks to see if the user-agent is Googlebot by looking for "Googlebot" in the user-agent.
2. Checks to see if the hostname the IP address resolves to has "googlebot.com" in it.

If a visitor meets both criteria, the function returns true; if a visitor fails both criteria, the function returns false.

To integrate this with your existing code, simply call the is_google() function at the appropriate point in your logic. For instance, let's assume you have the following code that displays paid content to a logged-in subscriber and a login/registration form to users who aren't logged in and/or aren't subscribers:

if( is_subscriber() && is_logged() ) {
    // show subscriber content
} else {
    // show login/registration form
}

To treat Google like a logged-in subscriber, change the first line to:

if( ( is_subscriber() && is_logged() ) || is_google() ) {

That's it. When the Googlebot pays you a visit, it will be able to access all of your juicy subscriber content.

Note: this isn't fool-proof. The "Googlebot" user-agent can easily be spoofed and by setting up a hostname with "googlebot.com" in it (eg. googlebot.com.somedomain.com), someone could trick you into serving up your content. For a more secure solution, you should use a regular expression to make sure that googlebot.com is the top-level domain (I've skipped the use of regular expressions to do this in the interest of keeping the example here as simple as possible).

Giving Non-Subscribers a Sample

Of course, the purpose of this technique is not to deceive users. It's poor form to let Google spider your subscriber content so that it gets indexed but to then show the users Google refers to you a subscription/login page.
Therefore, if you choose to implement this technique, you should be courteous enough to provide a snippet of the paid content so that the user can quickly determine if you've got what he or she was looking for. The Wall Street Journal, for instance, does this:

Not only are these types of snippets the decent thing to provide, they will boost your conversions.

There are a number of ways to generate the snippet. There are plenty of functions out there that you can use for taking text and chopping it down to x words or you could simply use PHP's substr function to do the job.

Depending on your goals and how your site is set up, you may only want to show the snippet to users who have been referred to you through Google. Using the code above as an example:

if( is_subscriber() && is_logged() ) {
    // show subscriber content
} else {
    if( eregi("google", $_SERVER['HTTP_REFERER']) ) {
        // show login/registration form *with* snippet
    } else {
        // show login/registration form without snippet
    }
}

Making Sure Your Subscriber Content Doesn't Get Cached

You're almost there. There's one last thing left to do: make sure Google doesn't cache your subscriber content, which would quickly defeat the purpose of this entire exercise. To keep Google from making your subscriber content available through its cache, add the following meta tag between the HTML <head></head> tags on pages that contain subscriber content:

<meta name="robots" content="noarchive">

Conclusion

Again, this is a grey hat SEO technique that, while used successfully by quite a few sites, still carries risk. That said, given the resurgence of paid content as a business model and the negative effects a pay wall can have on SEO, it may be worth a look depending on your risk tolerance.

It's my hope that Google will eventually offer a means for operators of subscription websites to have their content indexed (and marked as such) as it does with Google News, which would eliminate the need for owners of subscription websites to turn to cloaking.

Photo credit: mythwhisper via Flickr.

Patricio Robles

Published 28 May, 2009 by Patricio Robles

Patricio Robles is a tech reporter at Econsultancy. Follow him on Twitter.

2380 more posts from this author

Comments (13)

Comment
No-profile-pic
Save or Cancel
Avatar-blank-50x50

BFG 9000

Google already allow for this - they call it "First Click Free".

TTFN


BFG

about 7 years ago

Patricio Robles

Patricio Robles, Tech Reporter at Econsultancy

BFG 9000,


First Click Free, in my opinion, is utter nonsense.

As I understand it, Google expects the website to provide the content for free anytime Google is the referrer for a subscriber-only page. For obvious reasons, this isn't a viable solution because a savvy user would be able to access all content for free simply by using new Google searches as a 'proxy' for discovering your content.

I respect Google's right to dictate its policies but there's a good reason most people I know have not implemented First Click Free. It's simply insane. If Google wants to address this issue head on, they need to do what they do with Google News.

about 7 years ago

Avatar-blank-50x50

Michael Grover

This is very very dangerous.  I think you are better off displaying a keyword-rich snippet on an article launch page.

Also, I understand the First Click Free loophole of users turning to google as their primary navigation.   Do you know how it behaves on sites that use Google Site Search or the free version of google search?

about 7 years ago

Patricio Robles

Patricio Robles, Tech Reporter at Econsultancy

Michael,

Yes it is, which is why it definitely isn't for everybody. Hence my disclaimers.

That said, Google isn't doing anyone (itself included) any favors with its policy here.

It's frustrating to know that you could wake up one morning and find that you've been banned from Google, especially since there are sites that do this that have very high rankings. But if you have a paywall and want to be indexed, what other options are there to consider?

I don't know how FCF functions with Google Site Search. Would assume that it functions exactly the same. As long as the referrer is Google, you'll be displaying the content...

about 7 years ago

Avatar-blank-50x50

Clerkendweller

Your disclaimer is right on the mark.  These techniques wouldn't really be suitable if the content contains sensitive (e.g. personally identifiable information) or highly valuable data.  But you are correct, this type of approach may be appropriate for some organisations and some data.

I know it would be long-winded to include full code in this article, but server environment variable values like $_SERVER['HTTP_REFERER'] should be validated and if necessary sanitised before being used in scripts.  Don't trust anything!

about 7 years ago

Avatar-blank-50x50

dizi izle

Thank admin.

perfect system.it is Working

about 7 years ago

Avatar-blank-50x50

Mike

Been doing this for 10 years :)  Even told Google we were letting their bots in as paid members and they were fine with it.  All we say was Google news adding "subs" label to the site(s) tag.  Now Google news explicity allows cloaking for subs sites we just need the main search index policy to reflect reality. 

One click Free is an abhorrent mechanism for publishers that rely on fees for their income - it is just too easy to circumvent.  Allow dark-matter material to be indexed by the bot then give the site rights to give away content on their own terms.

about 7 years ago

Avatar-blank-50x50

Tony

So is it smart to do that or not? Can i be banned or not from search results?

almost 7 years ago

Avatar-blank-50x50

MyCelebrity.net

Same question like Tony, is it smart to do that?

almost 7 years ago

Avatar-blank-50x50

James Askew

Its so irritating that there is no black and white, right and wrong for things like this. I have been penalised for this offence and suffered dramatic loss in rankings, yet i had alot of content freely available alongside the member only content. After removing script and submitting a re inclusion request i got my penalty removed. So frustrating that some high profile sites get away with it and others dont.

over 6 years ago

Avatar-blank-50x50

Web Hosting

Is this really works? I guess google will terminate subcription cloaking. Can you assure me that I'm not going to be banned in Google?

Thank you

over 6 years ago

Avatar-blank-50x50

bio

perfect system.it working

over 5 years ago

Avatar-blank-50x50

Spartacus

Thans admin i tested it
Good works fun :)

over 5 years ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.