In August last year I wrote an article called Foundem vs Google: a case study in SEO fail. Foundem had been complaining about Google, due to its lowly search rankings.

My article was based on a story published in The Guardian, which pretty much sounded like a big bunch of sour grapes to me. As such I called out Foundem, which didn’t appear to be doing an amazing job of SEO best practice.

But Foundem insisted that The Guardian article had been heavily edited, claiming that the newspaper’s lawyers didn’t want it to use words like ‘penalties’ and ‘whitelisting’ in the article, when referring to Google. Big, scary Google.

IT WAS A PENALTY! OR WAS IT?

Foundem claimed that its inconsequential search rankings were due not to any SEO-related issue, as I had stated in my article, but the result of some hideous penalty imposed by Google. The reason for the penalty? The firm says via its Search Neutrality blog that it was penalised “simply because it is a vertical search engine” (as opposed to a technical or spam-related penalty). This seems thoroughly unlikely given that Google indexes thousands of other search-based companies.

A penalty would explain why the site wasn’t ranking brilliantly, but then again so would other things. As Fresh Egg’s Jaamit Durrani says: “Most of the time people claim ‘penalty’ when it is just a roadblock (eg iframes/javascript) or a filter (eg duplicated / scraped content). Often fixing these onsite issues can lead to immediate reinclusions.”

I asked Google to comment, and while it won’t talk about individual sites, it did confirm that it does not whitelist sites, nor does it punish sites purely due to their sector.

Google’s Peter Barron said:

“Our algorithms are designed to provide users with the best possible search results. We publish our guidelines for webmasters and advertisers, but we do not penalise specific types of site. 

“Vertical search sites are important to us and our users – indeed sites which offer added value often come top of our search rankings. We make all of our search quality assessments based solely on relevance to the user.”

None of which sounds like a sector-based ‘penalty’ to me, but then again…

YOU FOOL, YOU GOT IT ALL WRONG!

Earlier today I endured a fiery 35-minute call with Foundem CEO Shivaun Raff, who advised that: “On 1 December our penalty was lifted”, adding that it led to a 10,000% surge in traffic. And this apparently proves that my previous article “is incorrect from the headline down”, and should be corrected. 

I refuse to touch it, as I maintain that the Foundem site doesn’t tick enough best practice boxes in order to secure brilliant search rankings, and I’m not convinced that there was a penalty. Foundem says there was one. Google won’t confirm this, but says that it doesn’t punish sites simply because they operate in vertical search, as Foundem claimed.

Regardless, I stand by what I said, penalty or no penalty: there are enough reasons why the Foundem site might not rank so well. It isn’t a niche site. It contains tons of duplicate content, aggregated from various sources (Update: Foundem crawls and scrapes, as well as using feeds). There’s not much in the way of original content. It resembles a link directory. 

There were technical issues too, though some of these appear to have been fixed.

Jaamit Durrani explains: “Two of the major issues that Foundem had in summer was content in iFrames and content requiring javascript to load – both of which I looked at in August, and they were definitely in place. Both are huge barriers to search visibility in my book. They have been fixed somewhere between then and the lifting of the supposed ‘penalty’. I don’t think that’s a coincidence.”

Neither do I, though Shivaun Raff informed me – after a five-month delay – that this was one of the “mistakes” I’d made in my original article, and that the site is based on AJAX rather than javascript (I’m no techie but quickly pointed out what the J in AJAX stands for). 

In any case, there are many hundreds of vertical search sites out there, all competing for market share and for search rankings. Penalties aside, why Foundem merits a prominent placement hasn’t yet been explained to me.

YEAH, BUT BIG NASTY GOOGLE IS FAR TOO DOMINANT!

Ok, there’s no denying that Google is incredibly powerful. On this there can be no debate. Foundem points out that Google has a 90% market share in the UK, and 71% in the US. That adds up to a hell of a lot of control.

If you run a web-based business, like we do, then the failure to secure high search rankings can really hit hard. Many of us have experienced this directly (certainly we have). There used to be a saying that ‘if you aren’t on the first three pages then you might as well be nowhere’. Arguably, in any meaningful sense, it’s the first page or bust. It’s scraps after that.

There’s also the issue of Google using universal search to push its own products. These aren’t limited to the likes of Google Maps, YouTube, or Google News, but also vertical search products (e.g. loans, mortgages, real estate, price comparison). I personally don’t find this surprising, given that Google is in the business of search, and does it remarkably well.

Yet Foundem says that this “preferential placement” bypasses “the algorithms it uses to rank the services of others”. I guess it does. So why Google would need to ban or penalise sites in order to get a competitive advantage is beyond me…

Big, nasty Google.

OK, BUT WHAT ABOUT YAHOO AND MICROSOFT?

So yes, Google will link to its own products. Who wouldn’t? Certainly not the likes of tiny, lovely Microsoft! Or small, cute Yahoo! Or any of the other major search engines!

Actually no, roll that back. According to Guava’s Edward Cowell, Bing “heavily integrates Microsoft’s own travel comparison and review services. So clearly it’s a direction search providers are looking to follow”. If I were a Google shareholder I’d be appalled if Google wasn’t doing this kind of thing. As a searcher, if it gets out of control or impacts on quality then I’ll start using another search engine. That’s how it works, right?

It seems that the Google haters have such short memories. Back in the day Yahoo quickly polluted its search engine to such an extent that it totally lost sight of ‘search’. It cluttered its pages with all manner of proprietary services, becoming a ‘portal’ in the process. All of this attention-purging clutter served to distract the user from the core ‘search’ proposition. 

Google, on the other hand, has always believed that its homepage is sacred. It still does, and is in fact more minimal than ever. Just look at the recent fade-in feature, which presents the user with a search box and two buttons, and that’s it. Compare that with Yahoo.com and you might understand why it is Google that is synonymous with search, in the eyes of most web users, and not Yahoo.

The fact is that Google now dominates is simply because it does search very well, and its competitors have repeatedly failed to match it, at least as far as the user experience goes. And the user experience is incredibly important. Web users love the simplicity of the Google search experience.

And obviously the results must be of a high quality, otherwise searchers would quickly switch to some other engine for answers.

So yes, yes, Google is massive, but it’s massive for a reason. It didn’t get to where it is through nefarious means, but because it works better than any other search engine, at least in the eyes of the people who use it. 

YEAH BUT… WHAT ABOUT SEARCH NEUTRALITY?

Because big nasty Google is so big, we have a bit of a problem, and this is something that will intensify as the lobbying groups – included those funded by competitors like Microsoft – continue to make noises about Google’s market dominance.

I don’t want to gloss over the search neutrality debate, but then again Foundem said it didn’t want to have this debate on Econsultancy, preferring the New York Times to act as a mouthpiece (Shivaun Raff dodged an interview we’d scheduled back in October).

As such I’ll swerve on the navel gazing and will invite you to ponder a few questions, and to leave a comment if you fancy. There are no easy answers. I guess it comes down to two things: who do you trust most, and how might market dominance harm independent businesses?

Should Google and the other search engines be regulated?

Do you think that search engines should be forced to reveal their algorithms to some kind of search regulator?

Who should do the policing?

Do you want the government to own or significantly influence search?

Do you think all search engines should display the same results?

Is Google wrong to use its web pages to link to its own products?

My own view is pretty much based around free market principles. I’m not sure what the answers are, but the notion of state-policed search results is repulsive. Do you trust the government more or less than Google? I wonder what would George Orwell make of all this?