Agree with Daniel whoelheartedly about the session ID issue, but any form of IP delivery or User agent delivery is still classed as cloaking and frowned upon by search engines.

There are better ways of resolving this, including mod_rewrite, a host of scripting variations.

No company that uses dynamic content should experience difficulty with getting their content indexed.

Unfortunately, many of the content management system makers worked in splendid isolation when making their products and never looked at how the pages produced would be found by search engines.

This has resulted in many companies paying more than they need to to get decent traffic volumes to their site. Make sure you add this extra cost to the purchase price of the CMS.

Jim Banks
Web Diversity Limited
http://www.webdiversity.co.uk

On 15:38:02 8 January 2003 Daniel Phillips wrote:
>>>> Do search engines ignore URLs with
>querystrings
>>in them?
>
>>They don't ignore them, but they tend to treat them
>with
>>caution. The spiders are better at processing URLs
>with ?s
>>now, but can still get trapped (Scooter seems to be
>>especially prone to it).
>
>In our experience, search engines are getting better at
>dealing with URLs that contain query strings such as ?s.
>
>In the current search engine climate, Google is by far the
>most important in terms of market share - and Google seems
>to be indexing far more pages that contain these query
>strings.
>
>We have dynamic sites that have been live for only a few
>months and Google has spidered and indexed every page of
>these sites.
>
>What Google does not like are URLs that include session
>ids. These will stop the googlebot dead in its tracks,
>preventing further pages being spidered and indexed by
>Google.
>
>With some of our sites it was the case that human visitors
>using IE or Netscape would not see the session id in their
>URLs (as long as they had cookies enabled), but as
>Google's spider does not accept cookies, it would be
>presented with a session id in the URL. This would
>prevent the spider going any deeper into the sites,
>restricting the number of pages that Google would index.
>
>Using either IP Delivery or User-Agent Delivery it is
>possible to determine that the visitor to the site is a
>search engine spider, and to ensure that a session is not
>started. This will allow the spider to view all the pages
>of the site - seeing exactly what a human visitor to the
>site would see (except for any pages that require the user
>to log-in, etc.).
>
>This is not an attempt to 'con' the search engines, it is
>a method to ensure that they can see the same pages that
>human visitors can access. This is significantly
>different from IP Cloaking, which I would consider as
>seach engine spam. There is no attempt to try to present
>a different page to the spider than what a human visitor
>would see.
>
>Surely this is an ethical technique that doesn't attempt
>to 'cheat' the search engines. If Google were to question
>a site that their spider has crawled and viewed the site
>via a browser, they would see exactly the same as the
>spider. This has to be something that is best for all
>parties concerned - the website owners, the person
>searching on Google, and for Google itself.
>
>Regards,
>
>Daniel

I have just found an extremely useful website which helps to explain many of the mysteries of search engine optimisation, explains many methods of search engine "cheating" and explains why not to use them. It also goes into some depth about how Googlebot works.

Have a look at http://www.evolution7.com

BTW. There has been some discussion about using mod_rewrite to create crawler-friendly URLs. For IIS users, there are tools similar to Apache's mod_rewrite buy two companies http://www.qwerksoft.com and http://www.isapirewrite.com. I am currently trying to integrate the component by isapirewrite.com, and it seems to be working perfectly. It is allowing me to create complex maps from URLs to an asp page adding the required querystring data i.e.
/whitepapers/55.asp can be mapped to /whitepapers/view.asp?id=55. This is all done using Regular expressions, so brush up on your reg expressions...

I agree with your comment about commercial CMSs:
>Unfortunately, many of the content management system
>makers worked in splendid isolation when making their
>products and never looked at how the pages produced would
>be found by search engines.

The site I'm currently working on has a CMS which, as one of its features, adds a &SourcePageID=x to the URI. In other words, the URI reports the originating page. One side effect of this is inactivating "visited links", as you can have many unique URIs for the same page, depending on which page the link appears on.

This discussion made me realise that another side effect would be that the site could appear many times its actual size to crawlers, being indexed many times over and interfering with PageRank algorithms.

Does this sound like a valid concern? What potential benefit does the &SourcePageID information have? User journey analysis?

how valid is this post now? have things changed much since 2002?