In late 2009, Amazon introduced a new way for AWS customers to purchase its Elastic Compute Cloud (EC2) service: spot instances.

Instead of buying an instance outright at a fixed price, the price of a spot instance is determined by supply and demand.

So long as your bid for the instance is above the current spot price, you have a fully functional Amazon EC2 instance at your disposal.

For Amazon, the introduction of spot instances was an obvious solution to a costly problem: unused capacity. After all, Amazon has built one of the world’s largest cloud infrastructures, and if segments of it sit idle, Amazon effectively loses money.

For savvy AWS customers, this created an enticing opportunity. If the spot price for instances is typically below the price for on demand and reserved instances, it’s possible to lower the costs of using EC2, in some instances significantly.

There’s one caveat, of course: if the spot price frequently exceeds your maximum bid for spot instances, you’ll find yourself instance-less. When that happens, any processes running on spot instances are dead, leaving you in a bad situation, particularly if you’re running critical and customer-facing services.

Fortunately, there historically hasn’t been huge volatility in the market for spot instances, but when there is, it can be painful. Just ask SEOmoz. The popular provider of web-based SEO software is a heavy user of Amazon AWS, and relies on spot instances to reduce its costs.

The economic advantages aren’t hard to understand. As SEOmoz’s Brandon Forehand explains on the company’s developer blog, SEOmoz typically pays about 24 cents per hour for an instance that would cost 68 cents per hour as an on-demand instance. That adds up.

But recently, spikes in the spot prices of instances have caused major problems for SEOmoz:

However, we’ve been noticing quite a bit of pain the last week due to some extreme volatility in the spot price market. Normally, we use m1.xlarge instances, but over the last two days we’ve consistently lost machines in multiple availability zones due to the spot prices suddenly jumping to ridiculously high amounts.

How big were the spikes? Some took the spot price to a whopping $999 per hour!

According to Brandon, past spikes have typically been measured in minutes, and haven’t been anywhere near as extreme as the ones seen in the past week.

Obviously, $999/hour far exceeded SEOmoz’s bid for spot instances, so even though “we’ve always priced in a certain amount of volatility“, the company was left powerless as its spot instances were pushed offline, creating outages for customers.

Whether SEOmoz’s recent experience is a harbinger of things to come remains to be seen, but it does provide an important lesson for other companies using spot instances. The lesson: when you turn cloud infrastructure into a commodity that’s bought and sold, you have to recognize that there is downside exposure.

Playing in the market for spot instances is, in many respects, not that
dissimilar from playing in the stock and commodities markets.

One strong move against your position can wipe out all your gains, so investing most or all of your customer-facing services in spot instances is probably not a good idea, even if appears to be when things are going your way.

Instead, a more prudent investment strategy calls for using spot instances more sparinly, ensuring that they don’t make too much of your ‘portfolio‘, and that if you lose them, your entire portfolio doesn’t blow up.