SEO and database-driven web sites

Hi Warren

I read your feature on search engine optimisation in this month’s newsletter (http://www.e-consultancy.com/newsfeatures/newsletter/view.asp?id=361) with interest. I also read your piece on “Optimising dynamic web sites and database driven content” on your own site at http://www.greenlightseo.co.uk/optimising_dynamic_content.htm.

There does seem to me to be quite a challenge here and I’m not sure I’m clear yet on the answer. The challenge is: how do you optimise a database-driven web site for search engines? Let’s take two things as given:

1. Search engine optimisation is important and worth doing
2. Databases / Content Management Systems are the only sensible way to efficiently (cost & quality) manage large-ish web sites

If we take these 2 things as given then it is clear why the challenge is so important. I know I’ve thought about it a fair bit in relation to this site which is essentially one large database, some smart templates and plenty of business logic tying the two together to produce the front end. Pretty much nothing exists ‘flat’, it is all dynamic and most of it has to be – the personalisation features such ‘since your last visit’ clearly cannot exist hard-coded. Increasingly sites are seeing this kind of thing as important for improving the customer experience. So where does that leave SEO?

From all that I’ve read it seems that a simple, keyword-heavy flat HTML page with hard-coded links to other similar pages is best for getting top search rankings. This is not necessarily anathema to a ‘content management system’ – many CMSs can, or can only, publish flat HTML pages. Interwoven is a good example. It is a CMS and it is database-driven (at the development end) but does not produce dynamic sites. It outputs ‘flat’ sites. And there is a lot to be said for this, particularly in terms of performance (flat sites perform much better than ‘dynamic’ ones) and quality control (less chances of errors).

But let’s say we are talking about dynamic, personalised sites i.e. pages that don’t exist until the user requests them and they are then dynamically built. How do you optimise these? I guess there are 3 approaches:

1. They change: you wait until the search engines ‘upgrade’ their spiders so that they can spider such dynamic sites (your article says Google is making such steps)
2. You change: you restructure your site and change the way it works to accommodate the search engine’s spiders
3. You look for a work around

I would say that although most site owners recognise the importance of SEO it is not as important, say, as effective content management or perhaps an enhanced customer experience through personalisation. I would be very surprised, for example, if I was to be persuaded to alter the information architecture of e-consultancy (which is driven by business and user needs) to suit the needs of search engine spiders. However, this seems to be what your feature suggests? In your article on your site you say:

“Greenlight search engine optimisation usually tackles these obstacles at layer 1 of its multi layered optimisation programme, with what it refers to as the sub site layer. This deals with the technology on which a website is hosted, and in essence sets a foundation for subsequent optimisation.”

What exactly does this mean? Is it possible to accommodate search engine needs without compromising business and customer needs?

These issues have to date, I believe, left many site owners looking to option 3 above – a work around. ‘Doorway’ pages, which seem to be frowned upon, are a good example. I’m wondering, though, isn’t there a way to use CMS and personalisation technologies to our SEO advantage? For example, we could easily serve customised web pages to search engines’ spiders. We know ‘who’ the spiders are when they arrive so we could just then serve them specially optimised pages to help our rankings? This would not be ‘defrauding’ users or unfairly skewing search results, it would just help improve search rankings.

What do you think?

Thanks Brendon - that's very helpful. Do you know where I can find out more about these mysterious "XML trusted feeds". I can't find anything on Google about them, for example, though I have heard about them elsewhere. How much does it cost? What's the process involved?

Thanks

Ashley

Hi

You mentioned:
>1) You pay. For largish (min 500 page) sites, or clients
>of larger SEO outfits, there is a little know route to
>inclusion in some of the SE's databases called "XML
>trusted feed", which, when you get right down to it,
>is "legal" cloaking. The difference is that you
>are paying the SE's for the privelege of spamming them,
>so, magically, it's alright
How does one go about doing this? I have heard about this before, but am also not sure which of the main search engines support this, and how much it would cost.

You also mentioned the querystring ? in URLs effecting the indexing of your pages. Do search engines ignore URLs with querystrings in them? If so, would it be just as effective to offer a route for spiders to index which does not contain ? in the URLs, and a different one for users. Would this be ineffective as external sites would then be linking to the pages with the query strings in them.

If crawlers don't normally index pages with ? in them, do they not access pages within paged results i.e. 10 pages of results 1 - 10 ....

Thanks.
Matt

Hi Ashley

>>> avoid spamming
It is possible to optimise database driven sites without cloaking or cheating. Cheating using cloaking invariably ends up backfiring eventually as do many other forms of SEO spam, inspite of often impressive early results. Google invests much time in scaling anti cloaking and spamming solutions, and since they are the ones that know the algorithm, you are ultimately fighting a losing battle.

Getting any website indexed in search engines is down to it being able to be crawled, read and found relevant. DB driven, or not , dynamic or otherwise, if there is no way to crawl from one page to another via static links you will end up failing to be indexed, and even if you do get crawled, if your not the 1 of the 10 most relevant pages out of however many 100's, 000's or even 000,000's you still won't rank.

You need to develop crawl paths and ensure that your information architecture doesn't leave those essential pages of content which will leverage you the listings, locked up in a folder somewhere with no way to get to it unless the user performs some kind of action.

Search engine robots cannot perform any other action other than following html links. So no drop downs, no Javascript controls, no search functionality or anything involving an action. If the only way to get to a page is via one of these actions the search engine will have run into a cul de sac, your content won't get found, and they'll have nothing on you to match with user queries. Period.

Assuming reaching pages is not a problem you will then need to sort out your relevance for target terms. But we'll do that on another thread.

>>>As for Trusted feeds
What they allow you to do is insert custom page information specified in XML format directly into their searchable database, rather than have it indexed by spiders in the traditional and often detrimental way.

The advantages are that it allows you get large amounts of content indexed very rapidly in fairly controlled manner, and custom define what is displayed in the listings. We’ve used them very successfully on several clients now, and the traffic has been fairly impressive. Although given the engines that trusted feed’s target they may benefit consumer sites better than B2B.

The success of the exercise though lies in the creation of a clean and well optimised XML feed. Essentially its an inclusion tool, so including poor irrelevant content still won't solve your ranking problems. You need to make sure the information going in, is still relevant enough to yield a listing.

Currnetly Altavista, FAST and Inktomi have trusted feed programmes set up. Google won't do it as it undermines their alogorithm, and their very reason for being.

We have a complete Trusted feed, creation and management programme if your interested. More info at http://www.greenlight.co.uk/optimisation/cpc-seo.htm.

Best Regards,

Warren Cowan
Director, & Consultant SEOP
t: 0208 493 3780
m: 07958 764476
e:
w: http://www.Greenlight.co.uk

----------------------------------------------
Your total search engine marketing partner

Optimisation | Consultancy | Submission | Trusted Feed | PPC | Training | Web Analytics
Visit us at http://www.Greenlight.co.uk

On 18:56:27 11 December 2002 Ashley wrote:
>Thanks Brendon - that's very helpful. Do you know where I
>can find out more about these mysterious "XML trusted
>feeds". I can't find anything on Google about them,
>for example, though I have heard about them elsewhere. How
>much does it cost? What's the process involved?
>
>Thanks
>
>Ashley

>> Do search engines ignore URLs with querystrings in them?

They don't ignore them, but they tend to treat them with caution. The spiders are better at processing URLs with ?s now, but can still get trapped (Scooter seems to be especially prone to it).

You can get dynamic pages indexed, even top ten, but you need a powerful reason for the spider to visit (ie high PageRank or equivalent) and to offer them a method of identifying the vaild options (ie have all the options in a drop-down list, or similar)

Finding a way of eliminating the ?s is usually the best option, so use of mod_rewrite, or server-side rules, whatever. A competent web tech, or SEO will be able to advise you. Some dynamic systems that use ?s routinely often have a fix available, ie ColdFusion