AWS EC2 Reserved Instances sound like a great idea. You spend a lot with EC2, keep instances running for a long time and have predictable workloads throughout the year ? Reserved Instances to the rescue ! You pay a flat reservation fee upfront and reap the benefit of reduced computing cost over time.
All sounds great and makes perfect sense, until the day comes when you realize that the original forecast was off. For whatever reason. Perhaps you have improved the performance of your infrastructure and don't need as much resources. Unused reservations start piling up and you start bleeding money. Naively you say "not a problem ! we'll just sell these instances on EC2 Reserved Instance Marketplace and get our money back !". Well, that is where this story begins ...
There are two main problems with this kind of reasoning. First one is being able to accurately forecast future demand for resources from very limited historical information. EC2 reservations are usually sold at 1 and 3 year terms, while EC2 has really only been around for 6-7 years. And let's be honest, most of infrastructure you're trying to buy reserved instances for haven't been on AWS for more than couple of years. So, at best, you're really trying to forecast 1-3 year demand from 1-3 year worth of data ? Maybe you have a solid business, good plan that you won't deviate from and a strong reason to believe that engineering team won't be able to make things faster/cheaper anytime soon. But maybe you should think twice.
The second and more severe problem is liquidity risk you're running when dealing with EC2 Reserved Instance Marketplace. When people hear the world "marketplace" they imagine zillions of transactions a second, hordes of buyers, sellers and arbitrage-opportunity-seekers clearing the market in a blink of an eye, forces of supply and demand ensuring everyone gets the right price. But there is a huge difference between just saying something is a 'marketplace' and actually running a liquid one. In an inefficient market, it's very easy to end up on the short end ... and these reservations are not cheap.
RI Marketplace problems are not limited to simply not having enough buyers and sellers around. Market is highly fragmented - you can only buy and sell instances within the same region and can't really convert instance types. It is really a ecosystem of region-specific marketplaces, all of which highly illiquid, with each unique (region, instance type, duration, offering type) tuple representing a different commodity. There is no market maker either. You would expect that that Amazon might want to buy back your unused reservations at a discounted price, but that's not an option, at least not for now. After all, why give someone a fraction of their money back when you can keep all of it. To be fair, Amazon is making some efforts to 'improve' things - until recently you were only able to buy and sell instances within particular (region, availability zone) and now it's only (region) but the whole thing is still a long way from a real marketplace you can rely on.
So we went ahead and collected some data on EC2 Reserved Instance Markeplace activity over the last two months in order to get at least a bit more insight into how good or bad things really are.
Let's start with a simple marketplace snapshot breakdown (taken on Jan 9 2014). There are 44755 instances available across the entire EC2 RI Marketplace. That feels big ! Unfortunately, most of these are Amazon's own listings - for each (region, platform, instance type, availability zone, term, offering type), Amazon puts up their own "virtual" unlimited-quantity listing on the marketplace. So these don't really count ... and there are 44552 of them. Once we scrub that out, we come up with a much smaller number - only 203 listings on the marketplace. Now that doesn't necessary mean there are only 203 actual EC2 instances available, as some of these listings might be for quantities greater than one (AWS API doesn't provide that info). However, browsing around EC2 console, it is easy to see that the number of instances per listing is rarely greater than 10 (usually it's more like 1-2). So, quite a small number overall, way lower even then even just the number of different unique instance types.
Let's create a simple breakdown of instance types available for sale right now:
|Linux/UNIX, Linux/UNIX (Amazon VPC)||125 (61.5%)|
|Windows, Windows (Amazon VPC)||52 (25.6%)|
|Linux/misc (RHEL, SUSE)||12 (5.9%)|
|Windows/misc (SQL Server standard/web)||14 (6.8%)|
|Heavy Utilization||124 (61.0%)|
|Medium Utilization||42 (20.6%)|
|Light Utilization||37 (18.2%)|
The most popular combo seems to be (Linux, Heavy Utilization) in us-east (30 listings) and us-west (41 listings). So if you're looking at buying such reservations, you might be in luck. If you're looking at selling ... keep reading.
45-day Marketplace Dynamics
We now go back in time and analyze dataset consisting of 45-days worth of hourly snapshots of RI Marketplace, taken between Nov 28 2013 and Jan 9 2014. We plot number of instances available on the marketplace for each different breakdown:
We definitely see some, albeit relatively small action taking place. Occasional large dips & bounce-backs seem to be caused by someone taking a batch of instances off the market (in order to change the price, etc.) and returning them back on shortly afterwards. These we will ignore, but otherwise we'll assume that when instance disappears from the market, it has been sold.
With this assumption, we can plot the number of instances being sold vs. number of new instances entering the market throughout the given 45-day interval:
With this, we can now address the key question - how long does it take to sell an instance ? In order to determine that, we use simple Kaplan-Meier estimator, with event of interest representing the sale of an instance (listing disappearing from the market) and censoring defined in a standard manner (instance not being sold at the end of observed interval). Plot of obtained survival function estimates is shown in the following figure:
We can also estimate survival rate as a function of offering type (High, Medium, Low Utilization):
We notice that there is a significant difference between survival rates of different instance types, with Heavy Utilization having the lowest median time (14 days) and Medium Utilization the highest (19 days).
From given survival function estimates, we see that the median time-to-sell (for instances that eventually do sell) is 15.5 days and censoring ratio is quite high (30%). A very simple takeaway from this is - if you're planning on selling unused RI's, consider yourself lucky if you manage to find a buyer in less than a couple of weeks (in practice - anything less then a month is great). And if you're not selling something relatively popular (say, not a Linux/Unix in US east-west), it is highly likely that your unused reservations will expire before someone that needs them shows up.
Ironically, there probably are a lot of people that would benefit from reservations you don't need (and can't sell), but unfortunately AWS does not allow direct transfer of RIs between accounts. Not even for free. If this was to change, there would at least be an opportunity for over-the-counter deals (that is how illiquid assets are traded anyway), which might steer the marketplace in the right direction and make good things happen.
Until then, better think twice before going on a crazy RI shopping spree.