Mirchandani: Monitoring, analyzing, and optimizing costs have become an important part of managing your cloud environment, but it can be challenging to work with all your costs and usage data. On this episode of Beyond Your Bill, we'll look at exporting your billing data to BigQuery and how to create custom dashboards. Our last video covered how to explore your GCP costs with the billing reports that you can easily access through the cloud console. Now let's look at exporting it to BigQuery and then create a dashboard using Data Studio to further explore it. It's important to note that these actions can only be performed by a billing administrator. As we mentioned back in the resource organization video, you'll want to enable BigQuery export for your billing account as soon as possible. Ideally, you do this as soon as you create it. That's because the export will start collecting data as soon as you enable it and you can't get detailed billing data from before it's been enabled. With that being said, let me show you how to enable it. In the cloud console click on billing and choose the billing account you want to work with. You may have to click on manage billing accounts first then click on billing export on the left. Here's where you can see if the billing exports for BigQuery or file exports are enabled. While there is an option to do a CSV or .json export under file export, we don't actually recommend doing this because it doesn't contain as much information as the BigQuery export. The more information you have, the more analysis you'll be able to do. So go ahead and click on edit settings to choose a project and then choose a BigQuery dataset. If you don't have any projects, you'll need to create one with BigQuery enabled. Then create a BigQuery dataset. Creating a data set is as easy as going into the BigQuery interface, clicking on your project, and clicking on the create dataset button. If you already have a bunch of projects, feel free to make a new one just for this export or all of the other cost management efforts that you're working on. Once you know which project and data set you want to use, choose them in the export settings and hit save. You should see a screen that looks like this, which means that you've successfully enabled the export process. You can click the dataset name to jump into BigQuery, but you probably won't see anything yet. It may take a few hours to start seeing data since different services have different update frequencies. Also worth noting is that you may incur some nominal BigQuery costs for storing this data. You can quickly calculate any costs by looking at the BigQuery pricing page and seeing how much data you're generating every day with the export. You can also set a budget specific to BigQuery if you want to keep closer track of your spending. We'll cover this in more detail in a later video. Once the data starts showing up, you can run queries against it through the BigQuery UI. Here's sample query that shows the amount spent on each service by month since BigQuery export was enabled. This is a simple place to start, but there are lots of columns you can work with like location, project, and labels. From here you can build and run queries as you see fit to analyze your data or even start making predictions. Now let's talk about another way to analyze your billing data, creating a custom dashboard with Data Studio. Data Studio is an interactive analytics tool built into GCP and it can integrate easily with BigQuery to pull data and then let you visualize it. The quickest way to set up a dashboard is to start with the sample billing report that we've put together for you as a starting point. You can do this by visiting the sample billing report and clicking the how to copy tab. By following the steps outlined on that page, you'll set up some special queries and connect them to your own copy of the billing report in Data Studio. This simple report is ready to go with charts and filters to help you visualize your data. On the resources page, you can filter by label so you'll easily be able to see how your costs are broken down. For example, if you chose to label your resources by environment, you can see how much you're spending on production versus development environments. This is another reason it's important to choose naming conventions that are clear and easily understood by everyone. You're also able to build your own reports by using Data Studio. So I'll quickly show you how to get started with a basic report. First head to the Data Studio page and choose a blank report to start from scratch. Once it's been created look for the option to create new data source and then choose BigQuery. From here you just need to select the same project and dataset as well as the table that BigQuery created. After choosing it, click connect and then click add to report. Once the data source is added to the report, you'll be able to reference it with different graphs and filters. As an example, you can insert a new pie chart that shows how much you've spent on each service. Click on insert and then pie chart to add it to the report, then click on it to customize it. By changing the dimension to service.description and then the metric to cost, you can quickly see how your costs are split between different services. Different graphs have different fields, and you can create interactive elements to dynamically update them. Exporting your data to BigQuery lets you analyze your cost data using BigQuery and SQL as well as setting you up to visualize your costs with Data Studio or other data visualization tools. In the next videos, we'll look at advanced cost control tools like budgets and quotas.
