When Google purchased YouTube for $1.65bn in 2006, many questioned whether it could turn YouTube’s popularity into a big business.

Just over three years later, the answer appears to be ‘maybe‘. Google has made a lot of progress building an ad-supported business model for YouTube, but that alone might not be enough if YouTube is to realize its full potential.

In a post on the Official YouTube Blog, the YouTube team acknowledges that the ad-supported model has its limitations:

While YouTube has offered an easy and economical way for filmmakers — as well as content creators of all kinds — to instantaneously connect with fans around the world, many of them have told us that the ad-supported business model doesn’t always meet their distribution and monetization needs.

Fortunately, Google is paying attention and it’s finally getting set to experiment with another obvious business model that could bolster the bottom line for both YouTube and content creators: paid content.

This Friday, YouTube will give its users the opportunity to rent five movies online through a partnership with the Sundance Film Festival. Each of the five movies either appeared at the 2009 Sundance Film Festival or will appear at this year’s Festival.

But that’s not all. YouTube will be expanding the rental experiment shortly by offering “a small collection of rental videos from other U.S. partners across different industries, including health and education“. In addition, the world’s most popular video sharing website is seeking out independent filmmakers who want to rent their films on YouTube through a ‘Filmmakers Wanted‘ program.

According to the YouTube blog post, participating partners will have control over just about everything, from pricing to rental duration to viewing restrictions. Most importantly, they “keep 100% of their rights“. The technical details of how YouTube rentals will work for users were not revealed, although it appears that a Google Checkout account will be required.

Needless to say, paid rentals make a lot of sense for YouTube. Professional video content is increasingly finding its way onto the web, including through high-profile offerings like Netflix Watch Instantly. Certainly Google doesn’t want to miss out on this market.

The two big questions: how many content owners will oblige and can Google can build a compelling product for consumers?

Given YouTube’s massive audience, it’s likely that there will be no shortage of independent filmmakers and smaller content owners willing to give paid rentals on YouTube a punt. After all, some of them might make really decent money if Google turned even a sliver of its users into paying regulars. Larger content owners probably won’t come as easy, but that doesn’t mean they won’t come around, or that Google even needs them at the beginning.

Getting content is one thing but, in my opinion, Google’s long-term success in carving out a dominant position in this nascent market will probably depend most on the user experience. How easy will it be to checkout through YouTube? Will the quality be good enough? How will Google handle customer service? If Google has the right answers to these questions, rentals and other forms of paid content could eventually make YouTube’s $1.65bn price tag look like a bargain.