Newspaper publishers want the best of both worlds. They want the traffic that Google can deliver, but also think that the search engine owes them something. Specifically: “A fair share of the revenues being generated through the commercial exploitation of our content”.

Well, newsflash: Google owes the newspapers nothing. And now it has openly told the newspapers how to block web pages from the search engines by using the robots.txt no-inclusion protocol. If they want to, that is. There’s a barrier for those who want it. Now put up, or shut up. 

Of course this isn’t new to anybody, but Google’s stance, as paidContent puts it, “effectively raises a middle finger to the 169 signatories to the Hamburg Declaration on Intellectual Property Rights, including Dow Jones managing editor Robert Thomson and News Corp Europe CEO James Murdoch”. 

It’s going to be interesting to see if even one of those 169 signatories, or any other major newspaper, is actually brave, dumb and ballsy enough to take Google up on its offer. What’s the betting? I’ll wager that not one of them will go ahead with this. 

But, in the event that somebody out there wants to do this, then Google’s Josh Cohen helpfully explains its position, as well as how to go about it:

“News publishers, like all other content owners, are in complete control when it comes not only to what content they make available on the web, but also who can access it and at what price ...  if a publisher wants us to stop indexing their content, they're able to do so quickly and effectively.”

Cohen is pointing out the obvious, yet it will no doubt fall on deaf ears. These petty threats from the publishers are the desperate rantings of executives out of their depth. Is Google really to blame for poor forward-planning, sitting on hands, having no strategy, executing late or badly, and failing to diversify into emerging (and potentially profitable) new areas? I don't think so.

It didn't have to be this way

It was never going to be an easy transition. The media markets have fragmented beyond belief, driven by the sheer amount of choice and proliferation of devices through which you can access news, not to mention the way in which stories are distributed and consumed. But did it have to be this hard? Did it have to go this wrong?

Blaming aggregators like Yahoo News (four times bigger than Google News) and search engines like Google is madness. They help – rather than hinder - these publishers, by delivering unique users for free. Furthermore, a unique user that is referred from a search engine comes with some very important information attached: a search query in the URL string. You can see exactly what they searched on to find any given page. If publishers were smarter at selling audiences then this data would be incredibly valuable. 

As it turns out the publishing business seems to have hit some kind of race-to-the-bottom tipping point, by commoditising audiences, having allowed ad networks to sell out inventory at rock bottom prices. Truly, the power has shifted to the advertisers, but the publishers actively encouraged it to happen. They just didn’t care enough, and that’s why CPM rates have fallen so far, and why many media buyers remain lazy creatures that do not look beyond the numbers programmed into their speed dial. It’s a mess.

So what could they have done?

  • How about better content management systems and proper training for journalists?
  • How about a centralised, top-down strategy for local news sites, rather than ‘encouraging’ autonomy in satellite offices?
  • How about introducing the right kind of ad technology?
  • How about making better use of data, or even capturing and making sense of it to begin with?
  • How about dynamically optimising news sites in real-time for individuals?
  • How about diversifying their ‘media’ businesses beyond the classifed ads niche to create profit centres that underpin their news operations?

And so on and so on. I'm generalising a little bit here, but I have heard many horror stories from people working as journalists and sub-editors on national and local newspapers, not to mention the non-journalists, including some very senior executives. There are exceptions to the rule and fantastic things happening in some quarters, but equally there remain deep-rooted problems in the industry. 

Hindsight is beautiful

These things could and should have happened. Yet instead we get the accusatory comments from the likes of Trinity Mirror CEO Sly Bailey, who complains regularly about the ‘superdominance’ of Google. Some random quotes:

“That’s our copyright, and they don’t spend a penny on journalism at all - they just make money from ours.”

“We’ve been playing in to the hands of the very businesses that play so fast and loose with our content in the first place. We’ve become dependent on pats on the back from new kids on the block who tell us what the rules are.”

“People discover (stories) on Google, read it on our site and then flit away before discovering it was (us) who created it in the first place ... they click on an ad served up to them by Google for which we get no return.”

“Superdominant players like Google and the death of journalism as we know it.”

Is it just me? Google has nothing to do with the “death of journalism”. Absolutely nothing. If anything the newspapers themselves have dumbed down to such an extent that I have to second-guess myself before clicking on a headline. All too often AP stories are copy and pasted, with no differentiation between the top news sites that feature these stories. It’s truly appalling. Google should do a better job of burying these stories, which clutter the news aggregators and pollute the search results. It’s a low rent editorial decision because it's cheaper and it seems that there is no money to pay proper journalists any more.

And as such it’s a business model issue. And that has nothing to do with Google. If all of your competitors publish stories for free on the internet then that’s their decision, it’s an industry trend, and it’s something you need to figure out for yourself. Do you join them, or do you insist on payment of some kind, or make your site registration access? It's your call, but it's fundamentally an industry issue, not a Google issue. 

Rather than bleating on about Google maybe these newspaper publishers can group together to agree some kind of pact to start charging for content? Although if they keep republishing those AP / wire stories en masse then I for one would feel pretty hard done by, were I to pay to access it.

Put simply, Google is a distributor of content, where ‘content’ equals a headline and a short extract. By adopting the Robots Exclusion Protocol these aggrieved publishers can back right out of that distribution ‘deal’.

“Ah but is wasn’t a deal! We never agreed it! We demand a slice of Google’s ad revenue! Those bastards are making money off the back of our content!” 

Yes, well do you also want a fee every time I buy some chocolate along with a newspaper in my local newsagent?

Remember too that publishers are making money from the tens of millions of unique users Google sends to them every day. Like I said, they want the best of both worlds.

Publishers remain in a great position to generate revenue from digital channels. And there may be ways of working with Google that have yet to be explored, but to demand a slice of its ad revenue simply for being indexed is not something that's ever going to become a reality. 

Nobody said transition was ever going to be easy, but if you can know your audiences (as opposed to the singular ‘audience’) then you can monetise them.

The sooner publishers stop playing the blame game the better.

Chris Lake

Published 16 July, 2009 by Chris Lake

Chris Lake is CEO at EmpiricalProof, and former Director of Content at Econsultancy. Follow him on Twitter, Google+ or connect via Linkedin.

582 more posts from this author

You might be interested in

Comments (7)

Save or Cancel
Chris Lake

Chris Lake, CEO at Empirical Proof

I often read a bunch of sources for the same story. That's the effect of news aggregators and the immediacy of access to content. I do this to find out the story behind the story. The trouble these days is that there's a growing trend where everybody seems to simply publish syndicated content from AP / recycled press releases etc. The same story, almost word for word, across 'competing' news sites. It sucks.

The other thing is, most of the big publishers have internal SEO and social media teams that actively work at beating down the competition and optimising their presence on sites like Digg. They're feeding the very beast that they wish to slay (or at least eat a bit of).

And if Google's to blame, then where do we draw the line? Digg? Reddit? Does Twitter get a pass because it isn't yet making money / accepting advertising?

about 9 years ago



Does Google owe old media anything.  Probably.  Take away all old media and there isn't much in the way of news to aggregate.  All the' top 40 blogs' in the world aren't going to replace the Times' et al in most people's eyes.

However, is Google ever going to have to pay what it owes.  No.  Not unless all of old media get together in one massive bit of robots.txt collective action.

And as Wapping showed back in the day, old media ain't much of a fan of collective action.

However, just because Google (and the rest) are getting away with it, doesn't mean that they are innocent of all charges.

And as some old (forceably) retired print workers might say, "what goes around comes around".

about 9 years ago



But take away their access to the Associated Press and it wouldn't matter whether old media were indexed on Google or not because there would be precious little unique content to index. Cluetrain Manefesto…take a gander old media…

about 9 years ago


Chris Curtis

The bottom line is that the newspapers need someone to blame for their inability to keep up with the times.

If they took their content down, that opens a wider market for journalists to make money writing for online syndicates. It's the web - Someone WILL take their place.

A typical case of "If you can't beat 'em, just whine instead."

about 9 years ago


Ambarish Mitra

Publishers can't complain if google is driving traffic onto their site and making their content more popular. Content syndication does lead to finding the right source or original source for a breaking story.

Newspapers can always release a small excerpt of the story and publish the rest on a login or subscription basis. Only problem with this approach all stories is not logging worthy.

about 9 years ago



There is another aspect to all that, namely the pricing strategy newspapers

practiced. One more extreme example is the Rocky Mountain News, CO.,

now bust. It's demise was sort of clear since that article in 2004, telling

how circulation went down 11,6% while ad rates were raised a staggering

189% (in that period mentioned). The end came logically, Google had

probably not much to do with that.

A somewhat similar strategy is actually the case throughout. Now the

media are trying to apply every sort of magic in order to turn attention

away from even the most simple business calculation.

"Ad rates up cirulation down" can be used as searchword in case of further

interest in that topic.

about 9 years ago



I agree it's the newspapers themselves that have caused this rift.

They thought they could sit around on their hands while everyone else built the framework of content that we know today?

I also agree they create huge amounts of content. But they should have been at the forefront of monetizing that content, not whinging about it now.

Time for them to step up and make it work.

almost 9 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.