The FinOps Journey:
(EDITORIAL NOTE – Annotated in 2019 to show this post’s role on the path to the FinOps cloud operating model.)
One of the core concepts of FinOps is the creating of a central FinOps Team consisting of technology, business and finance team members. Among other duties, the FinOps Team is responsible for rate and discount optimizations, an idea with roots in the “centralize paying less” concept. At the same time, FinOps stresses the importance of empowering technology users to take an active role in rightsizing their use, a key part of the “decentralize using less” concept. Both of these have evolved to become part of the FinOps cloud operating model. Find out more by reading FinOps: A New Approach to Cloud Financial Management.
In the three months since I’ve moved to London to lead Cloudability’s EMEA expansion, a number of Britishisms have quickly become second nature, seeping into my US West Coast (and Hawaii-raised) accent. It’s amazing how fast decades of habits can change with enough regularity – faster, it turns out than the iOS autocorrect.
One decades-old habit that infrastructure people have is to let one team figure out how much capacity is needed and another team consume that infrastructure, often with little ongoing interaction between them. Cloud had changed that such that there are now a massive group of consumers of services in a company – all clicking buttons in the AWS or Azure consoles – who also need to become their own capacity managers. This has pushed ownership of the IT budget out to the edges of the organization. But some things still need to be centralized.
A Simple Formula for Spending
The formula for your cloud bill is really simple:
Spend = Usage x Rate
Usage might be the number of hours of a resource you used (or the number of seconds you used in AWS or Google’s clouds). The rate is the hourly (or per-second) amount you pay for the usage of that resource. Pretty simple. Increase either of those items and your cloud bill goes up. Increase both and it can go way, way up.
Two Levers to Affect Your Bill
Following that formula, you are left with two basic levers (properly pronounced “lee-vers” not “levers” I’ve learned) in order to affect your cloud spending. The first thing you can do is reduce usage. Our friend Mike Fuller, who runs cloud efficiency for Atlassian, calls this concept “Avoiding Costs.” In our upcoming 2017 re:Invent talk, he’s going to talk a lot about the ways you can do that. They range from terminating idle resources, to rightsizing oversized ones, to increasing elasticity.
The second lever (“lee-ver”) you have to pull is paying less for what you use. I call this rate reduction. Rate reduction typically means that you take advantage of cloud billing constructs in order to affect your cloud spending. Most commonly this is choosing Reserved Instances available from AWS or Azure. It can also be volume discounting based on usage or private pricing programs from providers who offer them. Some companies will also tackle Spot usage if they are willing to engineer around a potential sudden loss of the resource. Whichever way you go, they are all rate reductions or, paying less for what you do use.
Who Should Reduce Usage and Who Should Reduce Rates?
Once you recognise these two are the primary drivers of your bill then optimising looks a lot more simple. The debate begins when you try to determine who is responsible for each portion. Here’s my take, after 6 years of working with hundreds of companies who have done it poorly and a smaller number who do it really well:
Decentralize using less (avoiding costs), centralize paying less (reducing rates).
The people in your organization who are best equipped to make a decision about shutting things down or resizing them are going to be the application owners themselves. These people can look at a rightsizing report and determine whether that instance that appears to be idle needs to stay around or can be terminated. However, these people are typically not great thinking about what RIs to buy, nor do they always understand the mechanics of how RIs work.
Conversely, the best people in your organization for reducing rates would be a centralized cloud team (or person) who can look at the entire estate to find opportunities for savings. I affectionately call this person or team the RI Czar. These people are measured on metrics tied to increasing reserved instance coverage, reducing RI waste and taking advantage of volume discounts or private pricing.
Why You Should Centralize Rate Reduction
First, RIs are not purchased for individual instances, they are purchased as coupons that can be applied to any instance hour that matches their criteria. In many cases, this means that a single RI may be used across multiple instances depending on the hour (or second in AWS’ case). Likewise, RIs can float across tags or linked accounts, so it may be that account A “needs” 3 RIs but they can take advantage of 2 that are in account B.
Second, keeping RI coverage up and RI waste down requires ongoing management. The initial purchases can be complicated to get right (I’ve seen millions of dollars wasted in bad RI buys that looked at the wrong type or OS). Rather than having multiple teams run their own rate reduction efforts, centralizing the practice and process optimizes your team’s coverage.
All of the units of cloud resources that your teams expect to run at a reduced rate won’t all come from one team. As enterprises at scale run multiple products and projects that require hundreds to thousands of cloud resources, compiling all of the reservations into one centralized means of monitoring increases coverage.
For instance, if a front-end app team hardly uses any reservations for their small footprint, they shouldn’t reserve a year or three-years’ worth of reservations. Instead, they could “borrow” from the excess reservation portfolio of another team, like a back-end data team that might have extra reservations to help reduce the front-end team rates. Having one team manage all of these reservations, rather than all cloud-based teams running their own reservation portfolios, lowers the potential for waste while optimizing coverage.
Check out our free Reserved Instance e-book for a deeper look at the strategies behind cloud vendor discounts.
A few more high-level ways to centralize rate reduction include:
- Empowering a Cloud Czar or team to manage a “Waterline Strategy” for maintaining optimal reservation coverage
- Holding centralized rate reducers or Cloud Czars to KPIs related to cloud savings, such as annual savings on overall cloud, by department, etc.
- Implementing a repeatable approach to buying, measuring and learning from vendor discounts
I cover these best practices in great detail in a recent webinar. Check it out here and start at minute 33:00 for the details.
Why You Should Decentralize Usage Reduction
With rate reduction being strategically centralized, cost avoidance should be an activity promoted across all technology teams in your organization. Cloud costs at the enterprise scale can be comprised of hundreds to thousands of operations per day by various teams and various engineers supporting the services that your enterprise runs.
Empower teams by giving them the insight and data access to make the best technology decisions daily. This means incorporating the analysis and intelligence of a Cloud Optimization Platform (like Cloudability) into the rest of the metrics tracking that your Engineering teams likely already do. Whether it’s access into a specific dashboard set up for that team or an API call to integrate with the mission command that might already exist, engineering practitioners are given the right intelligence to make smarter cloud optimization decisions.
This practice isn’t just for technology teams. Business teams that rely on technology to support their objectives can get a custom view of performance and savings to help keep all members of your organization aligned with cloud optimization performance.
Whether your teams are incurring costs or viewing and managing them, everyone should be accountable for cloud optimization performance. This kind of accountability keeps costs in check and goes a long way toward optimizing cloud efficiency in an enterprise at scale.
Here are three key principles toward keeping cloud costs visible and increasing team accountability:
- Provide cloud cost data frequently: At whatever frequency fits the team, provide cloud cost and usage data and insights to indicate how teams are tracking to budget.
- Get granular: Show teams their spending filtered to their specific tags and accounts, and provide forecasting and alerts to indicate anomalies or significant shifts.
- Hold accountability: Track metrics closely and make progress visible to teams and their leadership.
Building and sustaining a strong tagging and allocation strategy is key to building the right data and insights to support enterprise-wide cloud optimization accountability.
I cover decentralizing usage reduction and related best practices in great detail in the same webinar referenced above (at minute 30:00).
Take Control of that Cloud Spend Formula
Remember, that’s: Spend = Usage x Rate.
When your teams have the platform and know-how to optimize usage and utilization and get the best rates on cloud resources, your enterprise will save at scale. Building the right cloud spend management strategy and respective tactics takes time and requires an iterative approach. To get started the right way, have the right platform to help your teams at hand.