Caveat emptor : Amazon EC2 Reserved Instance Marketplace

Caveat emptor : Amazon EC2 Reserved Instance Marketplace

AWS EC2 Reserved Instances sound like a great idea. You spend a lot with EC2, keep instances running for a long time and have predictable workloads throughout the year ? Reserved Instances to the rescue ! You pay a flat reservation fee upfront and reap the benefit of reduced computing cost over time.

All sounds great and makes perfect sense, until the day comes when you realize that the original forecast was off. For whatever reason. Perhaps you have improved the performance of your infrastructure and don't need as much resources. Unused reservations start piling up and you start bleeding money. Naively you say "not a problem ! we'll just sell these instances on EC2 Reserved Instance Marketplace and get our money back !". Well, that is where this story begins ...

There are two main problems with this kind of reasoning. First one is being able to accurately forecast future demand for resources from very limited historical information. EC2 reservations are usually sold at 1 and 3 year terms, while EC2 has really only been around for 6-7 years. And let's be honest, most of infrastructure you're trying to buy reserved instances for haven't been on AWS for more than couple of years. So, at best, you're really trying to forecast 1-3 year demand from 1-3 year worth of data ? Maybe you have a solid business, good plan that you won't deviate from and a strong reason to believe that engineering team won't be able to make things faster/cheaper anytime soon. But maybe you should think twice.

The second and more severe problem is liquidity risk you're running when dealing with EC2 Reserved Instance Marketplace. When people hear the world "marketplace" they imagine zillions of transactions a second, hordes of buyers, sellers and arbitrage-opportunity-seekers clearing the market in a blink of an eye, forces of supply and demand ensuring everyone gets the right price. But there is a huge difference between just saying something is a 'marketplace' and actually running a liquid one. In an inefficient market, it's very easy to end up on the short end ... and these reservations are not cheap.

RI Marketplace problems are not limited to simply not having enough buyers and sellers around. Market is highly fragmented - you can only buy and sell instances within the same region and can't really convert instance types. It is really a ecosystem of region-specific marketplaces, all of which highly illiquid, with each unique (region, instance type, duration, offering type) tuple representing a different commodity. There is no market maker either. You would expect that that Amazon might want to buy back your unused reservations at a discounted price, but that's not an option, at least not for now. After all, why give someone a fraction of their money back when you can keep all of it. To be fair, Amazon is making some efforts to 'improve' things - until recently you were only able to buy and sell instances within particular (region, availability zone) and now it's only (region) but the whole thing is still a long way from a real marketplace you can rely on.

So we went ahead and collected some data on EC2 Reserved Instance Markeplace activity over the last two months in order to get at least a bit more insight into how good or bad things really are.

One-Day Snapshot

Let's start with a simple marketplace snapshot breakdown (taken on Jan 9 2014). There are 44755 instances available across the entire EC2 RI Marketplace. That feels big ! Unfortunately, most of these are Amazon's own listings - for each (region, platform, instance type, availability zone, term, offering type), Amazon puts up their own "virtual" unlimited-quantity listing on the marketplace. So these don't really count ... and there are 44552 of them. Once we scrub that out, we come up with a much smaller number - only 203 listings on the marketplace. Now that doesn't necessary mean there are only 203 actual EC2 instances available, as some of these listings might be for quantities greater than one (AWS API doesn't provide that info). However, browsing around EC2 console, it is easy to see that the number of instances per listing is rarely greater than 10 (usually it's more like 1-2). So, quite a small number overall, way lower even then even just the number of different unique instance types.

Let's create a simple breakdown of instance types available for sale right now:

Region

us-east-1  79 (38.9%)
us-west-1  50 (24.6%)
us-west-2  37 (18.2%)
ap-southeast-2  11 (5.4%)
eu-west-1  9 (4.4%)
ap-southeast-1  9 (4.4%)
ap-northeast-1  7 (3.4%)
sa-east-1  1 (0.05%)

 

Instance type

t1.micro  38 (18.7%)
m1.small  29 (14.2%)
m1.medium  24 (11.8%)
m1.large  22 (10.8%)
c1.medium  20 (9.8%)
m2.2xlarge  13 (6.4%)
c1.xlarge  12 (5.9%)
m1.xlarge  11 (5.4%)
m2.xlarge  9 (4.4%)
cc2.8xlarge  8 (3.9%)
hi1.4xlarge  7 (3.4%)
m2.4xlarge  6 (2.9%)
m3.xlarge  2 (1.0%)
m3.2xlarge  1 (0.5%)
c3.large  1 (0.5%)

Operating System

Linux/UNIX, Linux/UNIX (Amazon VPC)  125 (61.5%)
Windows, Windows (Amazon VPC)  52 (25.6%)
Linux/misc (RHEL, SUSE)  12 (5.9%)
Windows/misc (SQL Server standard/web)  14 (6.8%)

Offering Type

Heavy Utilization  124 (61.0%)
Medium Utilization  42 (20.6%)
Light Utilization  37 (18.2%)

The most popular combo seems to be (Linux, Heavy Utilization) in us-east (30 listings) and us-west (41 listings). So if you're looking  at buying such reservations, you might be in luck. If you're looking at selling ... keep reading.

45-day Marketplace Dynamics

We now go back in time and analyze dataset consisting of 45-days worth of hourly snapshots of RI Marketplace, taken between Nov 28 2013 and Jan 9 2014. We plot number of instances available on the marketplace for each different breakdown:

Region

Screen Shot 2014-01-11 at 10.53.13 PM

Instance type

Screen Shot 2014-01-11 at 11.15.34 PM

Operating System

Screen Shot 2014-01-11 at 11.55.20 PM

Offering Type

Screen Shot 2014-01-12 at 12.16.50 AM

We definitely see some, albeit relatively small action taking place. Occasional large dips & bounce-backs seem to be caused by someone taking a batch of instances off the market (in order to change the price, etc.) and returning them back on shortly afterwards. These we will ignore, but otherwise we'll assume that when instance disappears from the market, it has been sold.

With this assumption, we can plot the number of instances being sold vs. number of new instances entering the market throughout the given 45-day interval:

Screen Shot 2014-01-12 at 12.55.16 AM

With this, we can now address the key question - how long does it take to sell an instance ? In order to determine that, we use simple Kaplan-Meier estimator, with event of interest representing the sale of an instance (listing disappearing from the market) and censoring defined in a standard manner (instance not being sold at the end of observed interval). Plot of obtained survival function estimates is shown in the following figure:

Screen Shot 2014-01-12 at 3.12.31 AM

 

records n.max n.start events median 0.95LCL 0.95UCL
581 581 581 405 15.5 14.4 17.8

 

We can also estimate survival rate as a function of offering type (High, Medium, Low Utilization):

Screen Shot 2014-01-12 at 3.17.44 AM

 

records n.max n.start events median 0.95LCL 0.95UCL
Heavy Utilization 344 344 344 235 14 11.7 16.7
Light Utilization 101 101 101 71 18 15.5 22.9
Medium Utilization 136 136 136 99 19 14.4 22.5

 

We notice that there is a significant difference between survival rates of different instance types, with Heavy Utilization having the lowest median time (14 days) and Medium Utilization the highest (19 days).

Conclusion

From given survival function estimates, we see that the median time-to-sell (for instances that eventually do sell) is 15.5 days and censoring ratio is quite high (30%). A very simple takeaway from this is - if you're planning on selling unused RI's, consider yourself lucky if you manage to find a buyer in less than a couple of weeks (in practice - anything less then a month is great). And if you're not selling something relatively popular (say, not a Linux/Unix in US east-west), it is highly likely that your unused reservations will expire before someone that needs them shows up.

Ironically, there probably are a lot of people that would benefit from reservations you don't need (and can't sell), but unfortunately AWS does not allow direct transfer of RIs between accounts. Not even for free. If this was to change, there would at least be an opportunity for over-the-counter deals (that is how illiquid assets are traded anyway), which might steer the marketplace in the right direction and make good things happen.

Until then, better think twice before going on a crazy RI shopping spree.

  1. I actually gave away (sold for $1) a tiny instance at US West. It took about two months to give it away. (Windows Heavy Utilization)

    I've listed my m1.medium Windows Heavy Utilization RI, also at 'give away' price of $1 and it's been more than two months with no takers.

    The situation is considerably worse than even you guys have estimated.

    As you put it, this 'so-called market' is pure fantasy. When I search as a potential buyer, I find exactly zero instances like mine for sale, so even though I have the ONLY one like it for sale and IT'S FREE! I can't get rid of it.

    I just checked eBay as an alternative but there are zero listing on eBay. (I was hoping to find a larger market; no such luck!)

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>