Research firm Yankee Group has removed a report on the virtualization market “after several inaccuracies were reported“.

VMware, a virtualization vendor, pointed out the inaccuracies in the report “which seemed to favor its competitor Microsoft.

The report was apparently written without the financial support of any vendor. Yankee Group has not provided any further information about how the report was compiled.

The incident highlights the fact that research firms, whose data and analyses many take at face value, are not always 100% accurate and should not be relied upon as such.

Based on my past critique of Forrester Research’s work in the area of social media marketing, readers will probably not be surprised when I state the opinion that much of the research sold by research firms is of questionable value.

The most recognized research firms, almost all of which are for profit enterprises, may technically be independent, but that does not mean that they always produce reliable, trustworthy information.

After all, as businesses they have a profit motive. From providing data about emerging sectors to analyzing the differences between industry solutions, research firms may often be free from overt bias (i.e. conflicts of interest with specific companies) but they do have bias – a bias to create the impression that their research is needed.

After all, it’s in a research firm’s best interest to find a reason to write reports that can be sold. If Forrester Research, for instance, looked at social media marketing, a hot topic that companies are talking about and willing to spend money experimenting with, and came to the conclusion that the hype was not yet backed up by substance, there would be a lot less research to sell.

My skepticism about research firms is borne out of the observations I made during Bubble 1.0. As a young lad working in the startup world and exploring businesses of my own, I inevitably came across research reports touting impressive projections for emerging markets and highlighting a wonderful world of opportunities.

After Bubble 1.0 popped and markets matured, I couldn’t help but notice that most of the impressive projections and wonderful opportunities promoted by the research firms never materialized. It was as if the projections had been pulled out of thin air and the opportunities touted disappeared into that same thin air just as quickly.

One need only look back at the wild predictions made during Bubble 1.0 to see that. In 2000 (the Webvan days), Forrester made bold predictions about the market for online grocery sales in Europe, predicting a €55bn market by 2005 with significant growth in uptake.

Yet Tesco, the largest British grocer with the largest online marketshare, “onlypulled in £748m in online sales in the first half of 2007 and a November 2006 Forrester report stated:

“Online grocery shopping struggles to reach a broader audience, with only 3% of online consumers buying groceries online in 2007 compared to 2% in 2002.”

This is a perpetual trend: just prior to Google’s IPO in 2004, a article stated:

Forrester predicts Google will cede ground to portals such as MSN and Yahoo as they improve their search technologies and begin to take advantage of customer loyalty gained through their Web mail services.

Ironic that just four years later, Microsoft is looking to acquire Yahoo, seeing the acquisition as perhaps one of the last vehicles to compete against Google in the search market.

To be fair, research firms like Forrester are tasked with an almost impossible task – predict the future. Nobody can. At best, educated speculation can be provided.

Even then, significant differences are bound to exist, highlighting the fact that even educated speculation is still speculation. Compare the huge variation in forecasts as to what the mobile ad market will be worth in a few years.

The big problem, of course, is that research firms are often looked upon as authoritative sources. Many investors are easily seduced by the projections that are produced by research firms and if a research firm predicts that the podcasting market will be worth $1 trillion by 2012, there are some individuals who can’t be convinced otherwise.

This is unfortunate, not only because I think that research firms are in little more than the business of intellectual speculation but because I increasingly question how they operate.

Beyond the fact that research firms have a financial motivation to produce plausible-sounding research that begets the production of more related research, it’s my opinion that serious boundaries are being straddled and possibly even crossed.

Specifically, I see researchers who are relied upon to provide objective information who have essentially become members of the industries that they research, jumping from conference to conference and forming what appear to be close personal relationships with other members of those industries.

Obviously, it would be naive to argue that researchers can’t have any “involvement” with the industries that they research but I personally believe that some of the involvement seen creates, at the very least, the potential for bias, even if unconscious.

For instance, I cannot help but note that Forrester’s social media reports are often written by the analysts who have published a book called “Groundswell: Winning in a World Transformed by Social Technologies” which purports to show companies how they can turn the “unstoppable” trend of “social technologies” from a threat into “an opportunity”.

Regardless of whether or not the ideas in Groundswell are valid, if I was a Forrester client, I would have to consider the fact that reports written by these analysts are reports written by individuals who reasonably could be perceived as having a vested interest in a certain perspective of the world.

If that perspective originated from extensive research they’ve conducted for Forrester, it is still hard to ignore that a financial interest (their book) gives them a possible incentive to maintain the same perspective, even if new data and research indicate that the perspective is flawed.

At the end of the day, I think those who read research reports would be wise to heed the following common sense advice:

  • Do not consider research firms to be authoritative sources of information, analysis and data. Just because a research firm says that Market X will be worth $500 billion in 5 years doesn’t mean that it will be. And just because a research firm says that you should engage in Activity Y using Application Z doesn’t mean that you should.
  • Look at who writes a report. What has the author written in the past? Has the author’s past advice stood up to the test of time? What is the author’s background? Does the author have vested interests that create room for potential bias – conscious or unconscious?
  • Conduct your own research, perform your own analysis and, most importantly, apply basic logic. There’s absolutely no substitute for any of these things.

All this said, I am willing to go out on a limb and make a prediction of my own – research firms will continue to make lots of money publishing “research” no matter how many questions are raised about accuracy, sensibility and plausibility.

Perhaps I simply need to publish a research report about research firms. I think the headline will read “Drama 2.0 Predicts Research Firms to Issue More Research; Fools to Buy Record Number of Reports Promoting Hype and Touting Fanciful Projections.

Sadly, I’m sure there are people who would pay $500 to read it.