When you use resources on Amazon Web Services, Amazon records your usage data and accompanying cost data in several different locations. In addition to the information available through the standard set of billing and cost management tools that AWS provides in the console, you can also access several different types of billing reports which provide varying levels of granularity into your cost and usage. Understanding what data is available through each of those different reports is key to knowing how and where to access different components of your cost and usage data, and is also useful in understanding how Cloudability pulls and relays that information.
The Billing Report is the most basic source of cost and usage information that AWS provides. It’s accessed through the AWS Management Console on the Bills page, under the “Summary” tab, and is also available as a CSV that can be published to an S3 bucket.
The report provides a summary of charges accumulated during a specified month, and details a breakdown of those charges by service and usage type. In a similar vein you can also access a list of totals for all accounts within a Consolidated Billing setup on the “Consolidated Bill Details” tab. Notably, this information only includes aggregate total, and does not break down spending by hour/day or by tag.
The Cost Allocation Report provides similar information to the Billing Report, with the additional delineation of spending by tag when tags are enabled. It’s generated as a CSV file, and once access is set up, publishes to an S3 bucket of your choice several times a day.
Like the Billing Report, the Cost Allocation Report provides a summary of charges accumulated during a given month, broken down by service and across accounts within a Consolidated Billing structure. The charges are reported as a month-to-date total for each usage type line item.
In the past, we based Cloudability’s daily cost and usage estimates on data pulled from the Cost Allocation Report. However, we now use the Detailed Billing Report with Resources and Tags for increased accuracy.
The Detailed Billing Report (DBR) is the second most granular data source provided by AWS. The DBR features a breakdown of all of your cost and usage across usage types for every hour. It’s generated as a CSV and published to an S3 bucket several times a day.
The DBR is unique in that it provides cost and usage data across your AWS infrastructure for every hour, rather than as a month-thus-far aggregate. Accessing this hourly data allows for a more comprehensive understanding of your usage behavior over time. However, it does not include tag- and resource-level granularity
The Detailed Billing Report with Resources and Tags (DBRRT) is the most detailed source of cost data AWS provides, and is the data source that Cloudability uses to access your cost and usage. The file contains the same data as the Detailed Billing Report, but with the addition of breakdown by tag and the specific ResourceIDs for the AWS resources used during the reporting period. As with the other reports, it’s generated as a CSV and is published to an S3 bucket with appropriate permissions a few times a day.
The DBRRT is by far the largest of the data files that AWS provides. Since it provides a breakdown of every single resource for every single hourly period of the month, the report becomes sprawling very quickly. For example, if you have two different instances running for an entire 30 day month, the file will use 720 lines (24 hours x 30 days) to detail the usage of one of those instances, and another 720 lines for the other—on top of additional lines for items such as data transfer, summary, and one-time costs.
Cloudability accesses your usage through the DBRRT by checking your designated S3 every few hours for the latest CSV. Each time a new DBRRT CSV is deposited in the S3, Cloudability pulls the data and updates what you see in the app accordingly.
The DBRRT includes 22 default items for each instance that delineate how that instance is being used, and which inform the values of Metrics and Dimensions that you see in Cloudability. These items include basics like “InvoiceID” and “PayerAccountID,” but also include such details as “UsageType,” “UsageQuantity,” “BlendedCost,” and “UnBlendedCost.”
Cloudability Cost Analytics and Reserved Instance Planner are powered by these items in the DBRRT. The Reserved Instance Planner, for example, scans every single line item in the DBRRT file, then summarizes the usage quantity for each instance class across the month and recommends RI purchases accordingly.
There are a few exceptions to the by-the-hour format of the DBRRT. These exceptions represent usage throughout the month and not by hourly period. One-time costs like RI purchases or support charges will appear this way, in addition to unused portions of Reserved Instances. These unused hours need to be factored into your bill—because you will have to pay for them, whether or not you use them. This part of the DBRRT is useful in determining whether you’re using your RIs, and is used to populate the Reserved Instance usage widgets within Cloudability.
You can access any of these reports as a CSV and drill into them yourself. However, particularly as you get into the more granular reports such as the DBR and the DBRRT, you’re likely to encounter a simply massive amount of data that can be hugely time-consuming to sift through.
While you can surface some valuable insight after working some Excel magic, you can spare yourself from spreadsheets by accessing the same data within Cloudability. To get started, log in or sign up for a free 14-day trial today.