{{ searchResult.published_at | date:'d MMMM yyyy' }}

Loading ...
Loading ...

Enter a search term such as “mobile analytics” or browse our content using the filters above.

No_results

That’s not only a poor Scrabble score but we also couldn’t find any results matching “”.
Check your spelling or try broadening your search.

Logo_distressed

Sorry about this, there is a problem with our search at the moment.
Please try again later.

In August last year I wrote an article called Foundem vs Google: a case study in SEO fail. Foundem had been complaining about Google, due to its lowly search rankings.

My article was based on a story published in The Guardian, which pretty much sounded like a big bunch of sour grapes to me. As such I called out Foundem, which didn’t appear to be doing an amazing job of SEO best practice.

But Foundem insisted that The Guardian article had been heavily edited, claiming that the newspaper’s lawyers didn’t want it to use words like ‘penalties’ and ‘whitelisting’ in the article, when referring to Google. Big, scary Google.

IT WAS A PENALTY! OR WAS IT?

Foundem claimed that its inconsequential search rankings were due not to any SEO-related issue, as I had stated in my article, but the result of some hideous penalty imposed by Google. The reason for the penalty? The firm says via its Search Neutrality blog that it was penalised “simply because it is a vertical search engine” (as opposed to a technical or spam-related penalty). This seems thoroughly unlikely given that Google indexes thousands of other search-based companies.

A penalty would explain why the site wasn’t ranking brilliantly, but then again so would other things. As Fresh Egg's Jaamit Durrani says: “Most of the time people claim ‘penalty’ when it is just a roadblock (eg iframes/javascript) or a filter (eg duplicated / scraped content). Often fixing these onsite issues can lead to immediate reinclusions.”

I asked Google to comment, and while it won’t talk about individual sites, it did confirm that it does not whitelist sites, nor does it punish sites purely due to their sector.

Google’s Peter Barron said:

“Our algorithms are designed to provide users with the best possible search results. We publish our guidelines for webmasters and advertisers, but we do not penalise specific types of site. 

“Vertical search sites are important to us and our users - indeed sites which offer added value often come top of our search rankings. We make all of our search quality assessments based solely on relevance to the user.”

None of which sounds like a sector-based ‘penalty’ to me, but then again…

YOU FOOL, YOU GOT IT ALL WRONG!

Earlier today I endured a fiery 35-minute call with Foundem CEO Shivaun Raff, who advised that: “On 1 December our penalty was lifted”, adding that it led to a 10,000% surge in traffic. And this apparently proves that my previous article “is incorrect from the headline down”, and should be corrected. 

I refuse to touch it, as I maintain that the Foundem site doesn’t tick enough best practice boxes in order to secure brilliant search rankings, and I’m not convinced that there was a penalty. Foundem says there was one. Google won’t confirm this, but says that it doesn't punish sites simply because they operate in vertical search, as Foundem claimed.

Regardless, I stand by what I said, penalty or no penalty: there are enough reasons why the Foundem site might not rank so well. It isn’t a niche site. It contains tons of duplicate content, aggregated from various sources (Update: Foundem crawls and scrapes, as well as using feeds). There’s not much in the way of original content. It resembles a link directory. 

There were technical issues too, though some of these appear to have been fixed.

Jaamit Durrani explains: “Two of the major issues that Foundem had in summer was content in iFrames and content requiring javascript to load – both of which I looked at in August, and they were definitely in place. Both are huge barriers to search visibility in my book. They have been fixed somewhere between then and the lifting of the supposed ‘penalty’. I don’t think that’s a coincidence.”

Neither do I, though Shivaun Raff informed me - after a five-month delay - that this was one of the "mistakes" I'd made in my original article, and that the site is based on AJAX rather than javascript (I'm no techie but quickly pointed out what the J in AJAX stands for). 

In any case, there are many hundreds of vertical search sites out there, all competing for market share and for search rankings. Penalties aside, why Foundem merits a prominent placement hasn’t yet been explained to me.

YEAH, BUT BIG NASTY GOOGLE IS FAR TOO DOMINANT!

Ok, there’s no denying that Google is incredibly powerful. On this there can be no debate. Foundem points out that Google has a 90% market share in the UK, and 71% in the US. That adds up to a hell of a lot of control.

If you run a web-based business, like we do, then the failure to secure high search rankings can really hit hard. Many of us have experienced this directly (certainly we have). There used to be a saying that ‘if you aren’t on the first three pages then you might as well be nowhere’. Arguably, in any meaningful sense, it’s the first page or bust. It’s scraps after that.

There’s also the issue of Google using universal search to push its own products. These aren’t limited to the likes of Google Maps, YouTube, or Google News, but also vertical search products (e.g. loans, mortgages, real estate, price comparison). I personally don't find this surprising, given that Google is in the business of search, and does it remarkably well.

Yet Foundem says that this “preferential placement” bypasses “the algorithms it uses to rank the services of others”. I guess it does. So why Google would need to ban or penalise sites in order to get a competitive advantage is beyond me...

Big, nasty Google.

OK, BUT WHAT ABOUT YAHOO AND MICROSOFT?

So yes, Google will link to its own products. Who wouldn’t? Certainly not the likes of tiny, lovely Microsoft! Or small, cute Yahoo! Or any of the other major search engines!

Actually no, roll that back. According to Guava’s Edward Cowell, Bing “heavily integrates Microsoft’s own travel comparison and review services. So clearly it's a direction search providers are looking to follow". If I were a Google shareholder I’d be appalled if Google wasn’t doing this kind of thing. As a searcher, if it gets out of control or impacts on quality then I'll start using another search engine. That's how it works, right?

It seems that the Google haters have such short memories. Back in the day Yahoo quickly polluted its search engine to such an extent that it totally lost sight of ‘search’. It cluttered its pages with all manner of proprietary services, becoming a ‘portal’ in the process. All of this attention-purging clutter served to distract the user from the core ‘search’ proposition. 

Google, on the other hand, has always believed that its homepage is sacred. It still does, and is in fact more minimal than ever. Just look at the recent fade-in feature, which presents the user with a search box and two buttons, and that’s it. Compare that with Yahoo.com and you might understand why it is Google that is synonymous with search, in the eyes of most web users, and not Yahoo.

The fact is that Google now dominates is simply because it does search very well, and its competitors have repeatedly failed to match it, at least as far as the user experience goes. And the user experience is incredibly important. Web users love the simplicity of the Google search experience.

And obviously the results must be of a high quality, otherwise searchers would quickly switch to some other engine for answers.

So yes, yes, Google is massive, but it’s massive for a reason. It didn’t get to where it is through nefarious means, but because it works better than any other search engine, at least in the eyes of the people who use it. 

YEAH BUT… WHAT ABOUT SEARCH NEUTRALITY?

Because big nasty Google is so big, we have a bit of a problem, and this is something that will intensify as the lobbying groups - included those funded by competitors like Microsoft - continue to make noises about Google’s market dominance.

I don’t want to gloss over the search neutrality debate, but then again Foundem said it didn’t want to have this debate on Econsultancy, preferring the New York Times to act as a mouthpiece (Shivaun Raff dodged an interview we’d scheduled back in October).

As such I’ll swerve on the navel gazing and will invite you to ponder a few questions, and to leave a comment if you fancy. There are no easy answers. I guess it comes down to two things: who do you trust most, and how might market dominance harm independent businesses?

Should Google and the other search engines be regulated?

Do you think that search engines should be forced to reveal their algorithms to some kind of search regulator?

Who should do the policing?

Do you want the government to own or significantly influence search?

Do you think all search engines should display the same results?

Is Google wrong to use its web pages to link to its own products?

My own view is pretty much based around free market principles. I’m not sure what the answers are, but the notion of state-policed search results is repulsive. Do you trust the government more or less than Google? I wonder what would George Orwell make of all this?

Chris Lake

Published 5 January, 2010 by Chris Lake

Chris Lake is CEO at EmpiricalProof, and former Director of Content at Econsultancy. Follow him on Twitter, Google+ or connect via Linkedin.

582 more posts from this author

Comments (12)

Comment
No-profile-pic
Save or Cancel
Avatar-blank-50x50

Ged Carroll

It is interesting that Foundem is a member of I-COMP: A Microsoft sponsored 'initiative for a competitive online marketplace'. http://www.i-comp.org/

over 6 years ago

Avatar-blank-50x50

JamieR

This is interesting stuff!
I've just read the NYT op-ed piece and I think your list of Search Neutrality questions at the end of this article is slightly incorrect. Looking at the principles of Net Neutrality the op-ed argues that Search Neutrality fits neatly under this umbrella. It's not arguing that search engines need to expose their algorithms, or that all search engines should display the same results, or that the government should own or influence search results. It is just arguing that search engines need to be open and transparent about their practices (just as the ISPs should be) and not to artificially demote some sites and promote their own.

This would obviously need some sort of arbitration: if your site isn't appearing in search results you should have the right to know why, if the answer is "because we don't think your relevant" then that is the result of that search engines particular algorithm. However if the answer is "because we think you're spam" or "because your are competing with our services" then you should have a right to reply to these accusations. At the moment not only does Google wield enormous power - there is no way to tell when, how and why it is using it - the answer to these questions is just silence.

As for google promoting it's own services above other results - I think this is clearly a case of anti-competitive practice. They are bascially promoting their own services above others. Even if you think that they are the most relevant, and that's what you want to see, that's only as a result of Google subsidising them using money earned from advertising - which should clearly not be allowed!

over 6 years ago

Avatar-blank-50x50

Malcolm Coles, Director at Digital Sparkle

The question for me is what terms Foundem thinks they should be ranking for, and why.

Looking at the site now, I see many of the problems I saw back in the summer: Overreliance on JS, pages that are uncrawlable without relying on form use ... etc

A typical example of why they don't rank

One example. If you want a Hitachi plasma TV, you have to go:

Fomepage > Electronics > Flat panel TVs > Plasma TVs > Plasma televisions - so 4 steps - and then use a drop down to select hitachi.

Only if you do all that with javascript off, there's no @££$ing button to press once you've selected the dropdown. So you actually can't get to the hitachi page.

It's just ludicrous. Their list of hitachi TVs is unreachable as a search engine. So if that's the sort of term they hope to rank for, it's no surprise they don't.

Even if it could be reached, their content about a particular model is standard, duplicate manufacturer content about each model. How is Google supposed to decide their identical manufacturer's description is better than everyone else's?

And then there's "Books etc"

Also, why is their top menu item called "Books etc" still? Tell me they're not after "Books etc" keywords on the ex-book retailer ...

And don't get me started on the books bit: http://www.foundem.co.uk/search/books.jsp

It's a search form. With no browsable links. What do they expect Google to do - continaully enter search terms so it can see if they have any books and then pretend those are normal crawlable pages?

On the other hand ...

I am, however, coming round to the idea that Google is a utility and needs regulating like one.

In particular, the business of the secrecy of its algorithm is a drain on everyone.

How much money and time is wasted trying to test and reverse engineer google's algorithms? Just think what we could have done better with the time and money we spent - are still spending over at SEOMOZ - blathering on about pagerank evaporation and nofollow ...

over 6 years ago

Avatar-blank-50x50

Alan Bleiweiss

A most excellent article, indeed Chris.  There are two different issues raised from my view.  First, Foundem clearly built a site that is extremely limited in it's ability to be properly indexed, and instead of investing thousands, probably tens of thousands of dollars to rebuild the architecture, they'd rather point fingers at Google.  This most likely gives them the fuel to justify to investors why they suck, so that investors keep feeding the dead horse. 

On the other issue, I do believe that while Google has, yes, kept their home page clean, the SERPs are really more polluted than ever.  This includes Google's own properties, but it also includes a lot of other crap as well.  I don't know how much more of this we'll need to deal with, however for the moment, Google's so dominant that most searchers will either put up with it or may, I'm afraid, like it, given the mentality of the masses.

Personally, I do agree with the line of thought that free market needs constraints, I just am not wise enough to have an answer as to what those should look  like or how they are supposed to be administered.

over 6 years ago

Avatar-blank-50x50

rishil

Really well written piece Chris.

over 6 years ago

Ciaran Norris

Ciaran Norris, Chief Digital Officer at Mindshare

I am, however, coming round to the idea that Google is a utility and needs regulating like one.

In particular, the business of the secrecy of its algorithm is a drain on everyone.

How much money and time is wasted trying to test and reverse engineer google's algorithms? Just think what we could have done better with the time and money we spent - are still spending over at SEOMOZ - blathering on about pagerank evaporation and nofollow ..

You're joking right Malcolm? That damn algo is just too hard to work out, they should be forced to reveal how it works? Maybe Coke should be forced to release their recipe/Tesco their CRM strategy/Simon Cowell his hair products?

over 6 years ago

Ian Miller

Ian Miller, Search Director at Crafted

As Malcolm says, their SEO is still shot.

For instance, Start in a category:

http://www.foundem.co.uk/search/computing.jsp

Title: Foundem - Search and Compare prices on Computers and Computer Peripherals

Then select any one of the options:

http://www.foundem.co.uk/G/computing/Computer-Software.jsp

Title: Compare Computer Software Prices at Foundem - Search Beyond Comparison

Now use the breadcrumb to go back to the parent category and the title tag (and H1) stay the same as the sub category. Regardless of the usability issues of things like this, with all the many other SEO issues they still have, this really isn't going to help them and as Jaamit said, many companies screaming "penalties" actually have poor sites.

Seriously, they're SEO is the problem, not Google. They're screaming at Google rather than getting their own house in order, because it's easier.

As for the whole search-neurtrality thing, I'm a believer in free market. It's Google's playground and they make the rules, much as I might not agree with them, if they aren't providing the best service then people will switch - albeit slowly.

Just because someone like Foundem start a price comparison site gives them no right whatsoever to be in Google's top results, especially when they are continually flouting many known (and quite simple) best practice rules.

Should Google and the other search engines be regulated?

To some degree, but no more than any other industry, my main concern is privacy and data mining more than anything about them promoting their own sites in search results. If Mark & Spencer owned a shopping centre, they'd be perfectly entitled to place their store within it, and in a good position. The other shops would be rented out to anyone else. If it wasn't what shoppers wanted, they'd visit another centre. They do need boundaries, but for me, promoting their own products is well within them.

Do you think that search engines should be forced to reveal their algorithms to some kind of search regulator?

Professionally, the more information that's out there for me the better. But then they shouldn't have to reveal their own IP, just as other industries don't. Plus, realisitically the more they reveal to anyone, however "secret" the more incentive for nefarious types to acquire it and the results will just get spammed, it's a big enough problem as it is.

Who should do the policing?

Other than general legislative contraints, the users, with their choices. As long as people know there are alternatives  then I'm happy

Do you want the government to own or significantly influence search?

No.


Do you think all search engines should display the same results?

Absolutely not. do you really want all shops to sell the same products etc?

Is Google wrong to use its web pages to link to its own products?

No, it's their search results and if they pollute them to the point of unsability they'll suffer just as Yahoo did.

Google isn't perfect, some of their recent "improvements" are anything but (real time spam, broken UKSERPS etc), but I think that to say that you have a right to be within their results at high positions (or to specifically know why not) is ludicrous.

over 6 years ago

Avatar-blank-50x50

malcolm coles

Ciaran - it may have been rather late at night ...

However, to stick with it for now: I don't think google is analogous to Coke. On the other hand, it is quite like Simon Cowell and Tesco in that it is an all-powerful gateway.

Leaving X factor aside, there are all sorts of calls for Tesco to be regulated / constrained etc because of its powerful position, and it's ability to roll over competitors and suppliers and dictate terms.

Should a search engine with 90% market share be immune from such considerations? Yes, it got where it is by being brilliant. Now it's where it is, examples like Foundem, Totlot etc do raise eyebrows / concerns.

And no, I don't think it should necessarily hand over the keys to the kingdon. But honestly, how much money and time do people waste wondering about second-link anchor text, nofollow and page evaporation etc etc. Can't they just tell us ...?

over 6 years ago

Avatar-blank-50x50

Shivaun Raff

Chris,

As I highlighted when we met in September last year and again on the phone yesterday, your first post contained a number of highly misleading inaccuracies.  Regrettably, you chose not to correct those errors, and your latest post only adds to them. 

There are far too many inaccuracies to go in to here, but I am particularly surprised that you repeat, and build on, your incorrect assertion that Foundem had “technical issues”.  Rather than simply recognising your original mistake, you instead compound the error by jumping to the erroneous conclusion that Foundem’s iFrame and Javascript infrastructure must have changed since your original post.  In fact, we have made no changes to this aspect of our site in the last year, let alone since August. 

Ultimately, it doesn’t matter that you are wrong about Foundem’s penalty (particularly since this penalty has now been lifted).  What does matter, however, is that your factually inaccurate posts continue to impede debate about the much more important issue of search neutrality.

Shivaun Raff

CEO and co-founder of Foundem.co.uk and co-author of SearchNeutrality.org

over 6 years ago

Ciaran Norris

Ciaran Norris, Chief Digital Officer at Mindshare

Malcolm - of course they could hand over the keys, but I don't think they should have to. It also doesn't mean that I think they should be immune from oversight or regulation, just that I don't think making them open the algorithm would be the way to do it, unless you make all the other engines, and any similar site do the same (should Amazon have to unveil its recomendation algo?)

Shivaun - you say potato... What are the phrases you were being penalised for? As far as I'm aware you've never said, and this detail would make it a lot easier for those of us who actually understand (at least a bit) how Google works to verify (or not your claims).

As for Chris impeding debate about more important issues, I actually think that the continued insistence of you and your colleagues to try and use your own problems with Google (which I sincerely feel are almost certainly self inflicted) to try and push this bizarre idea of search neutrality make it much harder to get to the real issues where the major engines are concerned.

over 6 years ago

Chris Lake

Chris Lake, CEO at Empirical Proof

Shivaun,

My original article sought to flag up areas for improvement on your site, to boost search rankings. Yes, it was critical, and no, it wasn't particularly focused on search neutrality. I explained why the site wasn't well-optimised, for your benefit, and for other people to understand what search engines like to see on a website. I stand by all of my points.

Both independent search experts and myself looked at your site in August and - among other things - we discovered all kinds of issues relating to iFrames and Javascript. Take another look at Jaamit Durrani's quote, mentioned above:

"Two of the major issues that Foundem had in summer was content in iFrames and content requiring javascript to load – both of which I looked at in August, and they were definitely in place. Both are huge barriers to search visibility in my book. They have been fixed somewhere between then and the lifting of the supposed ‘penalty’. I don’t think that’s a coincidence.”

If you are saying that you've made no changes in this area then I'm perplexed as to how the site has miraculously fixed itself. Fairies, probably.

So to be clear, you claim that you've made no changes to your site then, and that rankings only improved as a result of a penalty being lifted? Or have you made some changes?

You are also claiming that the penalty relates only to the fact that you operate a vertical search business, as opposed to some technical or content-related issue. Is that correct?

As to the notion that I am impeding debate, that's utter nonsense. A debate is a two-way street, and I have invited comment on search neutrality. But for me this isn't about search neutrality, it is about search optimisation, or the lack of it. If you don't dress for the occasion how can you possibly complain when you're not invited into the party?

Also, you might like to explain your involvement with the Microsoft-funded lobbying group I-COMP, as mentioned above by Ged Carroll?

over 6 years ago

Avatar-blank-50x50

Robert

Do you trust the government more or less than Google? I wonder what would George Orwell make of all this?

Fantastic questions.  I don't ever trust government, who knows what is really driving them?  Atleast we know with Google it comes down to the almighty buck.

I think that while the search engines offer differing results we can make up our own minds.  If they all served the same results, then it would be far easier for these to be open to manipulation.

While we still have options, we have some form of neutrality.

over 6 years ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.