The New York Times Magazine published an interesting article that discusses the Netflix $1mn challenge that has some developers working tirelessly to improve the accuracy of the Netflix recommendation engine by 10%.

The company's Cinematch technology is designed to introduce Netflix members to new movies that they're likely to enjoy. By 2006, the engineers at Netflix were unable to improve Cinematch's performance.

This posed an interesting challenge for the company because a substantial portion of its rentals were being driven by Cinematch. Today the company attributes a whopping 60% of its rentals to Cinematch recommendations. Most importantly, the company learned that Cinematch is driving more rentals, which helps Netflix retain its subscribers.

So what to do?

Netflix embarked on a crowdsourcing challenge - boost the performance of Cinematch by 10% and we'll pay you $1mn.

Today, 30,000 aspiring millionaires are trying to hit that elusive 10%. They've downloaded a list that contains 480,189 member ratings across 17,770 Netflix movies. No personally-identifiable or demographic data is provided.

The challenge has attracted a wide array of contenders - from Ph.D. computer scientists to self-taught hackers.

If anything highlights the importance of recommendations to online retailers, it's the Netflix challenge. While $1mn may seem like a hefty amount to pay for "only" a 10% improvement in a recommendation engine that already works quite well, Netflix knows that a 10% improvement will generate far more than the $1mn it has to pay.

In his article, which is well worth the read, the New York Times' Clive Thompson does a great job of taking a broad perspective of the online recommendation engines that are now so pervasive.

As Thompson points out, before the internet, if you needed a recommendation, you usually asked a friend, a family member, an acquaintance, a clerk. Or, of course, you relied on your judgment.

"The advent of online retailing completely upended this cultural and economic ecosystem," Thompson states.

Thanks to the fact that websites can track just about everything, it's possible for online retailers to come up with increasingly sophisticated ways to mine the vast amounts of data they collect to spot patterns that we could never detect on our own and to make recommendations that are quite accurate much of the time.

But as Netflix has learned, there do appear to be limits. The leader in the Netflix challenge has boosted the performance of Cinematch by 9.44%. Close, but no cigar.

According to Len Bertoni, who is a top 10 contender, the enemy of the Netflix hackers might be a single movie - the cult hit Napoleon Dynamite.

As Thompson observes:

"...'Napoleon Dynamite' is very weird and very polarizing. It contains a lot of arch, ironic humor, including a famously kooky dance performed by the titular teenage character to help his hapless friend win a student-council election. It’s the type of quirky entertainment that tends to be either loved or despised. The movie has been rated more than two million times in the Netflix database, and the ratings are disproportionately one or five stars."

"Worse, close friends who normally share similar film aesthetics often heatedly disagree about whether 'Napoleon Dynamite' is a masterpiece or an annoying bit of hipster self-indulgence."

According to Bertoni, Napoleon Dynamite accounts for 15% of his remaining "error rate." Given that he has been able to boost Cinematch performance by 8.8%, the Napoleon Dynamite "factor" is all that separates him from $1mn.

As I read Thompson's article, I couldn't help but think two things:

  • It's really quite amazing just how far we've come with recommendation engines like Cinematch. Although we take it for granted, the idea that we can now fairly accurately predict how well individuals will like, for instance, a movie, based completely on quantitative data alone is pretty amazing. The idea itself is not absurd (it's quite logical) but the success of practical implementations of it is really quite something.
  • There are, however, limits to recommendation engines. It may be possible for Cinematch to predict that someone who liked The Usual Suspects would likely enjoy Reservoir Dogs but quirky films like Napoleon Dynamite have more subtle, almost ethereal qualities that elude quantitative analyses.

We should appreciate the ability of computers and algorithms to add value to our lives (and to boost our businesses). At the same time, there are some things that just can't be explained by computers and algorithms.

I would suggest that we don't want computers to become so efficient at predicting our preferences and tastes that we never rent a bad movie again.

While I love technological innovation, there's a part of me that hopes the Netflix challenge won't be met. Perhaps if it isn't, it will have been the universe's way of telling us that our consumer culture isn't Newtonian in nature. That we can't mechanically predict what we're going to like or dislike. That there's an element of unpredictability that makes our lives a little bit more interesting and a little bit richer.

After all, if everything we consumed met our expectations, what fun would like be? What life-changing experiences would we miss out on? How would we grow? How would we learn? How would we shape our identities?

I find it somewhat ironic that society often pays the most attention to individuals who are dynamic and hard to "quantify." We generally have an appreciation for unpredictability. Who wants to do the same thing every day? Who wants to become so static that his friends and family can deduce everything about who he is and what he likes?

We're told that we should seek out new experiences, that we should try to learn something today that we didn't know yesterday, that we should challenge our beliefs. Indeed, we're taught to appreciate the fact that people can change.

Yet if we constantly endeavor to ensure that we never watch a movie we won't like, read a book we won't enjoy or listen to a song that hurts our ears, how will we ever maximize our potential as human beings?

While I think recommendation engines serve a very valid business purpose and certainly wouldn't recommend that online retailers do without them, the search for a perfect recommendation engine is really the search for a "positive feedback loop" that will ensure that we never consume anything we know we won't like because we're never exposed to anything that we might learn we like.

Bad movies are a part of life. While I really didn't really enjoy Snakes on a Plane, I will admit that I did smile once or twice. And who was ever harmed by a smile?

Drama 2.0

Published 27 November, 2008 by Drama 2.0

237 more posts from this author

You might be interested in

Comments (2)


Sam B

Having sat through the entirety of In The Cut starring Meg Ryan at a uni film club, I can say, hand on heart, that I would be quite pleased to never see a truly bad film again. I'm not talking about bad as in silly, or disappointing, or not-quite-my-thing, which is the kind of bad Snakes On A Plane rejoices in. I mean so bad you wish you'd walked out, and only stayed to the end in the hope that the heroine would die (she doesn't). Guy Ritchie's Revolver and City of Angels with Nicolas Cage are another two movies that I would have loved a robot to steer me away from. I didn't grow as a human while watching those films. There weren't any smiles. They didn't shape my identity. They were a waste of four hours of my short life. (I walked out of the latter two halfway through).

Since we're not forced to use recommendation engines, they can't possibly be the threat to our identity you see them as. "After all, if everything we consumed met our expectations, what fun would like be? What life-changing experiences would we miss out on? How would we grow? How would we learn? How would we shape our identities?" Well, part of what I expect out of life is life-changing experiences and personal growth. If I'm not getting those, then everything I consume isn't meeting my expectations, and I need to go out and find what will. If a recommendation engine keeps giving me excellent movies from the same genres I've liked for years, then eventually I will get bored with its recommendations and stop using it.

over 9 years ago

Drama 2.0

Drama 2.0, Chief Connoisseur at The Drama 2.0 Show

Sam: perhaps my pain threshold is simply too high. I like the good and the bad. The really bad just helps provide some perspective so that I remember to truly appreciate the really good. That said, I do avoid Meg Ryan movies.

I'm glad that you expect life-changing experiences and personal growth. I'm sure that you've have met a considerable number of people who expect neither and who have resigned themselves to an 8-5 life that they're not satisfied with. There's quite a few of them today and it's quite sad.

As for the comment that "we're not forced to use recommendation engines," this is, of course, true in theory. But in all practicality, recommendation engines of various kinds are pretty damn close to being ubiquitous. As Thompson noted, online retailing (and its associated recommendation engines) have "upended" part of the cultural and economic ecosystem that once dictated how individuals locate products and services.

I'm a believer that technology (Internet included) conditions people. There's some evidence of a link between text messaging and the degradation of writing skills. Studies have shown that the Internet is a disinhibiting medium that can impact individuals' interpersonal communications. And there's research suggesting that the Internet has contributed to tangible changes in the ways people "read."

Thus, it doesn't seem too farfetched that individuals (especially young people) who are conditioned by the Internet to trust recommendation engines for content discovery (over, say, personal recommendations or - gasp - experimentation) are liable to fall victim to what I see as being a positive feedback loop that these engines create. If individuals (consciously or unconsciously) are relying heavily on recommendation engines to choose products and services, eventually many (if not most) of them will increasingly be choosing from a smaller number of the same products and services.

A similar "positive feedback loop" is occurring in research. An interesting study conducted by the University of Chicago found the following:

"New research in Sociology shows that as more scholarly and research journals become available online, researchers are citing relatively fewer and newer papers."

"The Internet now gives scientists and researchers instant access to an astonishing number of academic journals. So what is the impact of having such a wealth of information at their fingertips? The answer, according to new research released in the journal Science, is surprising—scholars are actually citing fewer papers in their own work, and the papers they cite tend to be more recent publications. This may be limiting the creation and development of new ideas and theories."

"Evans, supported by the National Science Foundation, pursued the question by analyzing a database of more than 34 million articles. He used their online availability from 1998 to 2005 to compare the number of times the articles were cited between 1945 and 2005."

"The results showed that as more journal issues came online, the articles cited tended to be more recent publications that would have been cited without online access. Relatively fewer articles were cited, and citations concentrated in fewer journals and articles. 'More is available,' Evans said, 'but less is sampled, and what is sampled is more recent and located in the most prominent journals.'"

"Evans found a similar pattern across all areas of science and scholarship. Nevertheless, scientists and scholars in the life sciences showed the greatest propensity for referencing fewer articles, and it is less noticeable in business and legal scholarship. Social scientists and scholars in the humanities are more likely to cite newer works than other disciplines."

"Studies on how research is conducted show that people tend to browse and peruse material in print, but they search and follow hyperlinked references online. Digital search archives tend to organize results by date and relevance, influencing researchers to pick more recent articles from the most popular outlets. As they skim searched articles’ references online, they can follow them immediately, and much more quickly come in contact with articles others find important. As they factor others’ citation choices into their own, they become more likely to converge on the same articles and journals."

"Does this phenomenon spell the end of the literature review? Evans doesn’t think so, but he does believe it accelerates a process in which scholars and scientists come to a consensus and establish a conventional wisdom on a given topic. 'Online access,' Evans said, 'facilitates a convergence on what science is picked up and built upon in subsequent research.'"

I'd argue that recommendation engines promote a similar dynamic amongst consumers.

Variety is the spice of life and it's sad that we're seeing that diminish in so many aspects of our society today.

over 9 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.