We are currently helping a new client of ours develop a hybrid vertical search / content aggregation site and as part of the initial planning and consultancy phase we are looking at how the content is technically aggregated, categorised and tagged appropriately. As such we’ve come across a couple of issues which I believe could impact on the overall feasibility of the project and I am hoping someone with legal or relevant experience can offer some advice.

• Republishing content from 3^rd party sites if credited
  My gut feeling tells me that without express permission from 3^rd party websites, you are not legally allowed to scrape that content and then list on your own website for the purposes of search/browse/aggregation. However, if you consider all the search engines do exactly that where does one draw the line i.e. they will generally list extracts of the content rather than the complete content. Saying that, you can view the cache in Google search results which contains an exact copy of the content.
  Therefore, my question specifically is whether it is legal to scrape content from 3^rd party sites without any pre-agreed arrangements with them or even consideration of their terms & conditions so long as that content is publicly available on their sites? And if so, is there a ruling to say you simply need to credit them, or are there a set of laws about how you represent that content to ensure you are not presenting it as your own content but simply repurposing 3^rd party content for your site visitor?
   
• Robots.txt
  Following on from the point above, surely the mechanism which websites should use to restrict others from scraping and republishing their content in any form should be through the use of a robots.txt file which allows computers to communicate explicitly on what is or is not considered “crawlable”? Have there been any rulings surrounding that? 
   
• Monetisation of content
  Whilst our client is not proposing selling the content that they aggregate  for their users, they will be indirectly monetising 3^rd party content for which they do not pay any fees by including advertising, paid for services for advanced features and potentially even subscription services to access all content. Would this impact on whether they would be legally allowed to scrape and republish content? Google certainly don’t pass back any fees for the content they scrape...
   
• What happens if our client starts scraping and they subsequently ask our client not to do this?
  Assuming our client starts their business based on the fact they scrape and collate content from the few large players and the long tail of smaller players, and as a result their market prevalence increases to the point where one of the few large players (who currently monopolise the market) decide to expressly request that our client stops scraping their site, what rights or lack thereof would our client have? The risk that at any point their business could be hugely impacted by the fact that they are doing a better job than their competitors but at the same time rely on their competitors is clearly worrying.

Whilst I have a sneaking suspicion that the legality of scraping and re-publishing content on your own website is certainly in the favour of the publisher, I am surprised as I can think of many large players out there relying on a business model which scrapes content and repurposes it for various uses i.e. Google, Kayak, Kelkoo, Technorati.

What should I be advising my client in respect to their rights as a content aggregator, and how that may impact on their business in the future?

Thanks

Matthew O’Riordan
Digital Services Director
Aqueduct

Hi Felicity, Matt

In fact most news services would challenge you pretty quickly even if you did accredit them. Where they offer an RSS feed they expect people to use the title and story summary but then link directly through to their article on their site.

Some will allow you to publish the full article but generally at a cost.

Kind regards
Tony

hi, Matt, how's life?

I remember asking a similar question several years ago in a law lecture - there was no definite answer then. A couple of years later there was a test case where search engine caching was ruled 'fair use'.

Anyway, on to your problem...

If I was you, I would:

1. Publish snippets from scraped pages, with links through to the full version.
2. Include full content only from feeds, or content where you've done some kind of licensing deal.
3. Put a mechanism in place so that you could switch off particular sites from content scraping if they requested it.

In terms of robots.txt, I would:

1. Avoid crawling any restricted sites
2. Avoid publishing any content where any kind of 'no-cache'/'no archive' meta tags have been set (that's probably what's happening with the sky content you mention?)

I hope that helps you - would be interested to hear more about what you're  doing!

daniel

Matthew

You should look at the terms and conditions on the sites you intend to target.  You will also have to consider the liability for things like defamation, encouragement or glorification of terrorism, copyright theft and other intellectual property infringements, etc if you start publishing other people's content.  Take the advice of some good new media lawyers soon.

Also make sure that you don't start being used to circulate malware via infected content.

Some site owners are very agressive with those they consider are trying to take their content.  For example:

• Ryanair cancels aggregator-booked tickets in escalating scraping war
  http://www.theregister.co.uk/2008/08/13/ryanair_screen_scraping_cancellations/

Others use firewalling techniques to block scrapers, or to log what's going on for future court action.

Colin Watson
Technical Director
Watson Hall Ltd for website security