It's difficult to top AWS just based on an apples-to-apples comparison of buying servers versus turning up EC2 instances. EC2 is pretty price aggressive if you know what your needs are, identify an instance type that fits well, and reserve it for 3 years. A lot of people don't actually achieve all three, but it's often possible.
Most comparisons overlook the crucial price differentiator between AWS and a datacenter build: bandwidth costs. AWS bills based on bytes transferred, every IP transit provider bills by 95th percentile or similar.
A 1 gig commit on a 10 gig circuit is $1-2/meg in a well served on-net building, so let's say $2000/mo. The switch is $5000 or so, something that can't do full table BGP but is layer 3 capable, and support is $1000/yr. The cross connect is $300/mo. Plus a little bit more for optics and fiber. Over three years the cost is $91,000 (plus power to run the switch), if you never go above the 1 gig commit. Seems like a lot of money right?
Compare this to transferring 500mbit/s constantly to the Internet over 3 years at AWS pricing. That amounts to 156 TB / month transferred. Per month that will cost $11,878.40. Over 3 years the cost is $427,622.40.
There are some other key differences between AWS and datacenters:
- it puts all of your spending into opex, eliminating capex (this matters for some businesses);
- it limits the ways you can solve problems, there's no VRRP support for example, which is very limiting for a lot of service types;
- there is no ability to peer or receive settlement free transit if you deploy in AWS
However, in terms of raw dollars, the method that AWS uses to bill bandwidth consumption is always the major cost differentiator.
The bandwidth between AWS regions and Cloudfront is free so you can save a lot here as well while improving performance for your end users.
And on the Cloudfront side, you can negotiate pricing to further reduce the cost.
"If you are using an AWS origin, effective December 1, 2014, data transferred from origin to edge locations (Amazon CloudFront "origin fetches") will be free of charge."
Cloudfront is cheaper, that is true. The same 500mbit/s outbound via Cloudfront is $381,173 for 3 years at list prices. I think that this has given Cloudfront an enormous pricing advantage in the CDN market, considering that using any other CDN means paying EC2/S3-outbound plus the third-party CDN's prices.
If your content is highly cacheable, you can still use a third-party CDN and come out ahead. Since your EC2/S3 outbound will be 1/100th or 1/1000th of the total outbound, it starts to not really matter. For dynamic content, you're essentially being double billed, which is undesirable.
Either way, you're not going to get content out of AWS for an order of magnitude less money, which is what you can get in a conventional datacenter build. AWS is billing CDN-style: per byte transferred. When combined with the hit rate, this makes some amount of sense for a CDN service (like Cloudfront): every byte transferred has an impact on the edge caches.
I don't think that it makes much sense to bill EC2 bandwidth this way, but since people are obviously willing to pay it, Amazon has no real incentive to restructure their pricing.
This practice is probably the single largest source of lock-in for AWS, even above all of their hosted services like ELB and SQS. It's also why startups that run out of money sometimes can't let people take their data (photos or whatever): they simply can't afford the outbound bandwidth fees.
I don’t quite agree with all the calculations of AWS vs DIY.
m3.medium instance - 1 core, 3.75GB RAM, 4GB SSD
Typical server this days:
2*10 physical cores, 256GB RAM, 2 TB SSD
For around ~$10k
So you can run AT LEAST 20 m3.medium instances on a physical box (without overbooking). If you use overbooking a single server can handle probably 40 m3.medium class machines.
So instead of 100 servers, you need 30, or more probably 20.
Bottom line is:
People locked in the cloud mindset do not understand how fast physical hardware is this days compared to "cloud" offerings.
(storage): hs1.8xlarge: 16vCPU / 117GB / 48TB SSD @ $2.5740/hr (time to cost inversion with the 24 of the $10k servers needed to reach that density: ....10 years)
Hell, if storage is your thing, 48TB of 800GB Seagate 1200 SSD's alone would cost you $155,026 -- you wouldn't beat AWS's storage cost here until 6.8 years in. And that's not including the rest of the box to drive them.
> So instead of 100 servers, you need 30, or more probably 20.
Okay, then. $10k * 25 (split the difference, yeah?) vs. 100 * $.05/hr... $250k immediate outlay; you start getting cheaper than AWS (discounting labor, datacenter fees, and replacement hardware -- because I doubt your servers are going to last this long) in just under six years.
I'm always wary of posts that proclaim that one thing is definitively better, especially using overly simply comparisons. Cloud offerings and usage have very complicated billing and a huge number of line items. Many of these are very dependant on architecture and deployment choices and are very hard to predict up front. Add to that the complication that the people doing the deployments are often not optimising for price, it's a recipe for a big bill.
So on one level, I completely agree with you. In my experience, it's almost always cheaper to hand roll hardware yourself. That said, of all the reasons to move into the cloud, cost should be at the bottom.
Instead I feel companies should be asking themselves "Can I gain a competitive advantage by outsourcing my infrastructure?"
In many cases, the answer is yes - but it depends on the business; For example I can imagine that a business like Backblaze would not be able to compete as successfully using commodity cloud storage.
Yes, absolutely. Which means you need to architect and run expecting failure, which is so much different than traditional apps. (I am calling architectures 10 years old, "traditional"; that is funny.)
you're supposed to architect assuming failure no matter what. it's been like this for years. most engineers just choose to ignore best practices.
aws is just the first provider to explicitly tell their users "we don't care if your servers go down. tough cookies." which is what they all should have been doing in the first place.
I don't agree with the methodology for calculating AWS -vs- Moore's Law as mentioned in blog articles such as this one.[1]
A more accurate cost model would require multiple components in addition to number of transistors that amazon buys.
For example, to fully build an AWS service:
++You need a plot of land for the data center. Do real estate prices follow Moore's Law?
++Concrete and steel to erect the data center building. Do raw building materials follow Moore's Law?
++Energy costs. Does the price of terawatts follow Moore's Law?
++Bandwidth costs. Does the price of network transfers from Tier 1 backbones follow Moore's Law?
++Staffing costs. Do the programmers, system admins, and other techies' salaries paid by amazon follow Moore's Law?
++etc, etc.
If transistor count was the overwhelming cost item for supplying an "AWS" offering, we could then ignore all other component costs as an insignificant rounding error. Is this the case?
@jasode, actually, that is the exact point of the article. Non-Moore's-Law-subject costs are, at a bare minimum, 60% of the fully-loaded costs, probably more. All of that means that AWS costs should drop much slower than Moore's Law, about 60% slower. And yet, they keep dropping at a fast clip. Most of that would be due to Amazon's innovation and scale on everything beyond Moore's-Law-subject components.
It doesn't have to. If prices are constant and computers halve in size/power use every x months, real estate prices per bogomips halve every x months, too.
"Do the programmers, system admins, and other techies' salaries paid by amazon follow Moore's Law?"
Again, they don't have to. It also is possible to make efficiency gains. Here, I would guess Amazon has outsprinted Moore's law.
Similarly, I would guess network costs and speed have gone down faster than Moore's law.
Of the things you mention, my _guess_ would be that only energy prices are a significant factor stopping AWS pricing from following Moore's law.
But yes, it is more complicated than "pricing should follow Moore's law"
>Of the things you mention, my _guess_ would be that only energy prices are a significant factor stopping AWS pricing from following Moore's law.
Ah, but this is subject to Moore's Law too: As transistor density increases, performance per watt goes up (well, this requires Dennard Scaling also). Hence, as Moore's Law advances, the power costs go down (at constant performance).
No, but Google Compute Engine does a bit. For instance, Azure is at least twice as expensive than Google for cloud compute. (In addition to having their weirdass PaaS design leaking into their IaaS. Nice service otherwise.)
I was super hesitant to use Google for anything (A: because I dislike them now, B: I couldn't be sure they'd be committed to it, C: I recall they screwed people on Google App Engine?). But their offering is so much more straightforward and much cheaper. No commits, just use VMs and get discounts.
GCE is also way faster to start up. And single-core performance blows away Azure.
Azure and AWS make a big deal about storage and transfer, while ignoring they're way overpricing the CPU.
Nice story, but the numbers don't add up at all. If you order managed colo at a provider like Rackspace (which is a much higher level of support and guarantees than AWS, basically 24x7 and 100% on network and cooling etc) and get dual or quad CPU machines (really standard, I think they don't even have single CPU as an option anymore) with 256 or 512 GB ram you can run 64 or even much more "AWS medium" nodes on 1 piece of hardware. Because the AWS "core" is based on a very old type of Xeon.
So in reality you could run this setup with fully managed network and a 100% SLA on 9 or 10 dedicated machines at Rackspace (or even a cheaper competitor of theirs). The price for that would be around 9x 800 and the cost of two dedicated firewalls for a redundant high performance setup. So I would guess 7 to 8k USD per month or 288.000 USD in three years. Which is over 60% cheaper than the proposed AWS setup.
So maybe if you buy way too much hardware and combine it with a lot of manual work that could be done cheaper by a good provider, then yes... AWS looks cheaper. But if you look at a realistic setup on that scale, hardware is much more cost effective.
The sweat spot for AWS is a small setup or very flexible load. If you're big with a steady load, a smart setup with a managed provider is much more effective.
(And as a last remark: both AWS and Google have huge discounts for long commitments, Google even does it without upfront commitment. So if you use those prices the two options end up much closer again)
Quick comment on the comparison of AWS cost to DIY cost - for AWS costs you're using the on demand pricing for 3 years and comparing it to the cost of buying hardware over 3 years. If you're going to be running instances for 3 years you're probably going to be using reserved instances - in the case of m3.medium instances the rate drops from $0.070 / hour to $0.0261 / hour.
So, with the 3 year reserved pricing your total cost ends up being something like $411,544 - less than half the cost of the referenced $880,000 hardware purchase price.
You are 100% correct. That is why the article explicitly listed it as an assumption, and that the advantage for public cloud over private hardware is even better once you account for that.
Fascinating topic - yes AWS pricing falls slower than Moore's Law, this reflects the degree of market power shared by few large public cloud providers given the scale needed to compete at the top level. The profit maximizing strategy for each is enjoy rising margins while costs fall and reset prices to a lower level as a pack once costs have fallen so low that it is most profitable to lower the price to serve a larger market (google 'oligopoly' or 'kinked demand' for further clarity).
There was actually a slide addressing this very question used in a Urs Hölzle keynote at Google Cloud Live 3/25/14. It was titled 'but prices are not falling fast enough' and showed 2006-2014 cloud prices falling 6-8% vs. 20-30% improvement in hardware pricing. I included a screenshot in my deep-ish dive I'm developing on the economics of cloud market pricing http://www.stackalpha.com/blog/2015/2/25/cloud-price-wars-th...
Power - per kW cost would not have fallen materially, but efficiency per instance would have improved
Land - flat fixed cost would not have fallen, but sunk cost on existing facilities no marginal cost only relevant to geographic expansion. Also, AWS would own so would be working from increasingly small amortized base
Maintenance - might improve marginally with scale and experience as improve leverage on physical infrastructure
Staffing - would expect headcount per instance to fall in the period and automation to simplify some roles, offset by rising wage
R&D - You are 100% correct this would most definitely be increasing in the period and illuminates an interesting nuance. I am probably perverse in thinking of the underlying 'core' service (compute, storage, etc) margins as funding R&D to compete by creating the suite of services on top
For Google's purposes, this is immaterial. They already have core competencies in power, land, maintenance, staffing, R&D, etc that are sunk costs. Their point is that Amazon is enjoying 80-90% profit margins, and so this is a lucrative business for them to enter. Moreover, they're pointing out that they can offer you large cost savings sustainably to entice you to switch.
For a small business's perspective, the cost of overhead is absolutely material - but then, most of them probably wouldn't switch to bare metal anyway, but rather Dockerize their app and shop it around to whoever the lowest-cost IaaS provider is.
Claiming that a 20-30% reduction in raw hardware costs means you should see the entire service's cost go down 20-30% ignores those non-hardware costs, though.
Excellent. I like the numbers - it took a long time to hunt down numbers, the GigaOM article would have helped - but it is the underlying costs beyond the hardware that make it so.
The increasing energy cost in computing these days should change our thinking on Moore's law. We shouldn't be worried as much about doubling the power of a cpu, especially in a cloud setting, because you can always do that by using 2 cpus. We should instead create an equation based on the computing power, energy consumption, and energy prices.
remember, price != cost. pricing has many other influences.
Interesting conundrum however that the author discusses, that the gigaom link does not. At first glance this may appear that big consumers would be better off with their own hardware long term since they could theoretically follow the moore curve, but the missing piece is the cost of ownership AND cost of staying on the curve(since the curve keeps falling, you're continuously upgrading)
Next up is to build such that you can treat all cloud hosts as a commodity. Continuously monitoring pricing and loads and moving large amounts of instances from one provider to the next to optimize your costs.
I hope to have time to build a google spreadsheet to compare your numbers more closely.
Mose people don't realize that an AWS "core" is a threaded core. So the example quoted below of 2 processors with 10 cores is not 20 equivalents to amazon cores, but 40 cores.
Additionally, you're not depreciating your costs over several years as near as I can tell.
Whatever these numbers come out to, it's clear that if you're small and need unpredictable agility, AWS is cheaper. If you're larger and have enough foresight into your usage pattern, AWS is never cheaper at the right economy of scale.
> Whatever these numbers come out to, it's clear that if you're small and need unpredictable agility, AWS is cheaper. If you're larger and have enough foresight into your usage pattern, AWS is never cheaper at the right economy of scale.
Isn't that pretty much just a rule of the economy? Something like "It will always be cheaper to do it yourself, at a sufficient scale"?
It costs roughly the same to build an openstack cluster to server 600 m3.mediums as it does to use AWS. The openstack cluster is not better, but it is good enough. As you add more compute, the DIY solution becomes significantly cheaper, although probably not better.
The original poster used a silly piece of hardware, with no disk, 8GB of ram , and by far the most expensive CPU (in $/CPU) possible, with 18 cores.
The spreadsheet assumes a sane server with 128G of ram and a 300GB disk, which can fit 34 m3.mediums.
I assume you can hire a consultant at $10k/mo on a 3 year contract to spec, install, and operate this system, which, by the way, fits in 1 rack.
I assume you buy a 1gbps internet link for $3k/mo; and you have a budget of $10k for the switch, which should include maintenance.
These numbers show it would cost $222k in capital; which depreciated over 3 years (or do a 3 year lease) would cost around $6200/mo v. the AWS cost of 30k/mo with on demand.
Add $10k for the consultant, and you're at $16k v. $30k (AWS).
Assume a 50% discount for using Reserved Instances, and the solutions break out about even.
But, notice that 2/3 of your cost is that consultant at $10k/mo. So when you grow larger than the 600 VMs, things get cheaper as marginal costs increase.
There's no question AWS is valuable and the right solution a lot of the time.
The OP made some properly wrong assumption in how to build a hosted compute system, which, when corrected, show a much better cost comparison.
My numbers are also imperfect, but not as imperfect.
>, AWS is never cheaper at the right economy of scale.
Of course, it's a case-by-case analysis of whether AWS makes financial sense. However, I think some people have a misunderstanding on what the "right" economy scale could be.
The smart guys at Netflix (including Adrian Cockcroft) crunched the numbers it and it made more sense to go 100% on AWS rather than spend money on trying to maintain their own datacenters. Is Netflix a small time startup? Did CEO Reed Hastings properly evaluate that his business was the "right" scale? Your answer implies that if Netflix acquires X million more subscribers (whatever "X" happens to be), they should bring everything back in house and run their own bare metal servers. Is that correct?
The people that I know that work(ed) at Netflix say they should bring it back in house. My understanding was that Reed+team got snookered by IBM and said 'never again.' To wit, look at their CDN initiative, that's not outsourced to AWS.
Agreed that case-by-case makes sense, but we're pattern matchers, and we should group those cases into situational models.
1/ Adrian C. rocks. I had his sun performance book back in the 90s and think it and he is great.
2/ the "should being it back in house" is to save money. They love the abstraction and the in-house methodology and tools; but recognize that they're paying more to AWS than it would cost them to do it.
>but recognize that they're paying more to AWS than it would cost them to do it.
Adrian C came to a different conclusion.[1] He said it was cheaper to iterate & fail on AWS than on their own hardware. It was cheaper to expand into foreign markets by using AWS datacenters already there. It was cheaper to do large-scale fallback and disaster recovery with AWS.
I don't know who you talked to at Netflix but in my experience, there's an interesting division of opinion on whether AWS makes financial sense. In general, the tech team (admins, etc) think the in-house team can do it cheaper. But the business execs running the numbers in Excel see that AWS makes more sense. First the company tries things out with sandbox, dev, and qa on AWS and they get seduced by the speed and flexibility. (Why are we paying the IT team that takes 5 days to provision a dev box for the programmers?!?!) The business execs see possible value in moving more critical production workloads to AWS. The IT team that tries to keep it in-house can't show a convincing spreadsheet that calculates bare metal+staff is cheaper.
Most businesses' internal devops team (including the world-class one at Netflix) cannot match the pace of accumulating infrastructure expertise at AWS. This shouldn't surprise us since most businesses don't consider the babysitting of rack servers to be their core competency. I'm not convinced that Reed Hasting's internal team could execute faster and innovate cheaper than amazon's team. Sure, Netflix internal bare-metal servers will perform better than EC2 instances but there's also a speed to staff execution. Being slower on the admin tasks can negate the savings on the raw cpu hardware.
I'm always amazed that anyone can accurately calculate any cost of a set of Amazon web service elements, must less do it sufficiently accurately to measure over time.
Moore's Law refers only to the number of transistors in a component, and while transistors generally translates to computing power, this is not a 1:1 thing.
Lately the shift has been from CPU power to GPU power, so while there's more compute capability than ever, you need a hybrid system to take full advantage of Moore's Law.
It probably doesn't if the article's headline has to phrase it as a question instead of just stating outright "Amazon Web Services Pricing Follows Moore's Law".
GPU instances still seem overpriced. Reserved gets you $0.30 an hour, which would run you ~$8000 over 3 years. Hard to imagine the specs are much better than a $1500 computer.
The GPU instances use Nvidia GRID K2 cards, which cost $3000 in qty 1. AWS only runs two GPU instances per card, so they need to recoup the cost from fewer instances than typically share components. Nvidia's high end cards (GRID, Tesla) also have significantly disproportionate per-gflop costs.
The g2 instance costs have also come down a bit in the last 6 months, so they used to be even more overpriced.
For $8.4mm you can run 592 g2 instances reserved for 3 years and get 1,355,680 gflops. In a datacenter environment (including power, network equipment, installation, redundancy, etc.) you could achieve that level of performance for $3mm (not using GRID or Tesla cards). And then bandwidth prices really blow EC2 out of the water.
Most comparisons overlook the crucial price differentiator between AWS and a datacenter build: bandwidth costs. AWS bills based on bytes transferred, every IP transit provider bills by 95th percentile or similar.
A 1 gig commit on a 10 gig circuit is $1-2/meg in a well served on-net building, so let's say $2000/mo. The switch is $5000 or so, something that can't do full table BGP but is layer 3 capable, and support is $1000/yr. The cross connect is $300/mo. Plus a little bit more for optics and fiber. Over three years the cost is $91,000 (plus power to run the switch), if you never go above the 1 gig commit. Seems like a lot of money right?
Compare this to transferring 500mbit/s constantly to the Internet over 3 years at AWS pricing. That amounts to 156 TB / month transferred. Per month that will cost $11,878.40. Over 3 years the cost is $427,622.40.
There are some other key differences between AWS and datacenters: - it puts all of your spending into opex, eliminating capex (this matters for some businesses); - it limits the ways you can solve problems, there's no VRRP support for example, which is very limiting for a lot of service types; - there is no ability to peer or receive settlement free transit if you deploy in AWS
However, in terms of raw dollars, the method that AWS uses to bill bandwidth consumption is always the major cost differentiator.