Skip to main content

GCP Integration Guide

The following guide covers the steps to integrate your Google Cloud Platform (GCP) data with Vega Cloud.

Pre-Requisites​

The below Google Cloud Service APIs need to be enabled for the Google Cloud Project that will contain the BigQuery Billing Dataset.

You will need to enable the following Google Cloud Service APIs:

  • Cloud Resource Manager API (cloudresourcemanager.googleapis.com)
    • Reads and updates metadata for resource containers.
  • Compute Engine API (compute.googleapis.com)
    • Read VMs and volumes.
  • Cloud Billing API (cloudbilling.googleapis.com)
    • Ingest billing data programmatically.
  • BigQuery API (bigquery.googleapis.com)
    • Ingest BigQuery data.
  • Recommender API (recommender.googleapis.com)
    • Returns service recommendations for cost savings, security, and optimizations.
  • Cloud Storage API (storage.googleapis.com)
    • Allows interactions with Cloud Storage Buckets.
  • Deployment Manager V2 API (deploymentmanager.googleapis.com)
    • Deploy resources from a configuration.
  • Identity and Access Management API (iam.googleapis.com)
    • Manages identity and access control for GCP resources.

To enable these APIs choose a method below choose one of the two methods below.

Google Cloud User Interface (web browser)
  1. Navigate the Google Cloud Console to the library of APIs.
  2. On the Google Cloud Console home page, navigate to APIs & Services > Library.
  3. Search for the API you want to enable.
  4. In the API Library Search box, enter the name of the API you want to enable. Then press Enter to execute the search.
  5. Enable the given API.
  6. In the list that appears, click the API name to navigate to that API page. If the API is not already enabled, click Enable.
  7. After you enable the given API, the console displays a details page for that API.
  8. Repeat this step for each of the Service APIs listed above.
  9. Navigate to the console Home page.
  10. For each API you want to enable, navigate back to the home page and repeat these steps.
Google Cloud CLI (gcloud)
  1. Execute this command to enable the necessary Service APIs
gcloud services enable cloudresourcemanager.googleapis.com compute.googleapis.com cloudbilling.googleapis.com bigquery.googleapis.com recommender.googleapis.com storage.googleapis.com deploymentmanager.googleapis.com iam.googleapis.com

Step 1: Create Vega Custom Role in GCP with Minimum Permissions Sets​

Learn to create different roles as required with requisite permissions in GCP. Later those roles will be assigned to principals.

Vega Custom Role​

Creation of roles is very much dependent on your requirements. You can either create custom roles for various responsibilities such as monitoring, synchronizing, and then assign the required permissions to these roles. Or you can create a single custom role and assign all the permissions to that role. In our case, we create a single role. Later you can assign that role to the account whose service keys you will use to integrate with Vega Platform.

The Vega Platform uses billing data tables available within the BigQuery dataset to display the various cost reports. To perform the required actions, the role needs the permissions listed below.

Vega Provides a permission file containing all required access.

gcp-vega-admin-role.yml provides read only access to your Billing Account and any projects you assign the service account to.

Important: To create the Vega Admin custom role, you must have the iam.roles.create permission. By default, the owner of a project or an organization has this permission. Users who are not owners, including organization admins, must be assigned either the Organization Role Administrator role, or the IAM Role Administrator role.

Creation of custom role in GCP​

Create a custom role and assign permissions using one of the two methods below.

Please note that if you do not create the Role at the Organization level then the permissions must be added to the 
service account’s principle under **all** the projects linked with a billing account if not created at organization level.
Google Cloud User Interface (web browser)
  1. Log in to your GCP platform.
  2. Click "Activate Cloud Shell" near the top right-hand corner. (You may need to approve any requests related to this session.)
  3. In another tab, browse to the github link provided above, and copy the yaml code using "Copy Raw File" button. Switch back to the GCP tab.
  4. With focus in Cloud Shell: open a new file by typing nano vega-permissions.yml then press "Enter" key. Nano editor will open a new file.
  5. Use Paste (Ctrl+V) function to place the copied code into the nano file buffer.
  6. Press the key combination (Ctrl+X) to enter buffer-save mode. Press "y" to display the new file name, and "Enter" again to save the file vega-permissions.yml. Nano will close.
  7. Configure and run the syntax from line 8 below if you are using an organization to manage GCP projects. If you are not using an organization, you will be required to deploy permissions at folder or project level. To do this skip to step 9.
  8. Replace ORGANIZATION_ID (recommended) in the following syntax to create the role within your organization: Organization Role Creation - gcloud iam roles create VegaPlatformRole --organization=ORGANIZATION_ID --file=vega-permissions.yml
  9. Replace PROJECT_ID (not recommended) in the following syntax to create the role in a single GCP project: Project Role Creation - gcloud iam roles create VegaPlatformRole --project=PROJECT_ID --file=vega-permissions.yml
Google Cloud CLI (gcloud)
  1. Download gcp-vega-admin-role.yml to your local machine
  2. Create the Role at the appropriate scope for your environment - either at the Organization level or the Project level
    1. If you are creating the Role at the Organizational level then replace ORGANIZATION_ID with your Organization ID
      gcloud iam roles create VegaPlatformRole --organization=ORGANIZATION_ID --file=gcp-vega-admin-role.yml
    2. If you are creating the Role at the Project level then replace PROJECT_ID with the Project ID
      gcloud iam roles create VegaPlatformRole --project=PROJECT_ID --file=gcp-vega-admin-role.yml

Create Vega Billing Service Account​

Use Google Cloud's Deployment Manager to manage the resources necessary for Vega to efficiently collect Billing Data from your GCP Project. By deploying persistent resources (Cloud Storage Bucket, etc) using GCP Deployment Manager we are able to bind IAM Roles to the Service Account while limiting the scope of access to the resource in question.

The Deployment Manager Resource Template gcp-billing-resources.jinja deploys the following resources to your Project.

Resource Template NameResource NameResource TypePurposeNotes
vega-billing-service-account-{{ env["project_number"] }}vega-billing-sa-{{ env["project_number"] }}iam.v1.serviceAccountUsed to access BigQuery Billing DatasetsName is templated to ensure uniqueness
vega-billing-datasetVegaBillingDatasetbigquery.v2.datasetContains GCP Billing Data for your OrganizationThe Role roles/bigquery.dataViewer is granted to the Service Account for this Resource, limiting the Service Accounts access via this Role to this particular Dataset
vega-billing-export-bucket"vega-billing-export-{{ env["project_number"] }}"storage.v1.bucketUsed to export BigQuery data without incurring BigQuery Compute chargesThe Roles roles/storage.objectUser and roles/storage.admin are granted to the Service Account for this Resource, limiting the Service Accounts access to this Storage Bucket only

These steps can be performed using gcloud locally or in the Cloud Console available in the Web UI.

  1. Download gcp-billing-resources.jinja to your local disk (upload to Cloud Console if using that)
  2. Replace DEPLOYMENT_NAME with the desired Deployment Name to use (lower-case only!) - gcloud deployment-manager deployments create DEPLOYMENT_NAME --template gcp-billing-resources.jinja
  3. Once the Deployment has completed navigate to the Deployment Manager in the Google Cloud Console
  4. Select the Deployment from your list of Deployments, click on "Overview - DEPLOYMENT_NAME" to view the Deployment properties
  5. Find "Layout" in the Deployment properties and click "View"
  6. Copy the contents of the Layout panel and provide them to your CSM or Onboarding Engineer, the values needed are the "finalValue"s.
  7. Replace PROJECTNAME with the Project and SERVICE_ACCOUNT_EMAIL_ADDRESS with the _full email address of the Service Account created previously, this can be found in the Layout output. Run this command to bind the Big Query Job User Role to the Service Account created as part of the Deployment - gcloud projects add-iam-policy-binding PROJECT_NAME --member=serviceAccount:SERVICE_ACCOUNT_EMAIL_ADDRESS --role=roles/bigquery.jobUser --condition=None
Note: Step #7 above is necessary to avoid granting additional permissions (Roles) to the Google APIs Service Agent.

Step 2: Enable Cloud Billing Data​

Required Permissions​

You must have the following permissions to enable and configure Google Cloud billing data export to a BigQuery dataset:

  • Billing account administrator role for the target Cloud billing account.
  • BigQuery User role for the Cloud project that contains the BigQuery dataset that will be used to store the Cloud Billing data.
  • Project Creator role on the organization or folder. To create a new project, you must have the following permissions:
    • resourcemanager.organizations.get
    • resourcemanager.projects.create

Enable Cloud Billing Data to BigQuery​

Select or create a project that will contain your dataset.

  1. In the Google Cloud console, select or create a Project to contain your BigQuery dataset. (Must have Billing enabled, and be linked to the same Cloud Billing account containing data for export to BigQuery.)
  2. With project selected in "Select a project" drop down at the top of the Google Cloud Console: Open the Navigation menu > Billing. Depending on your scenario, you will see one of the following:

Projects with No Linked Billing Account

  1. Enable Billing on the project by either selecting Link a billing account or Manage billing accounts options.
  2. Continue to Create a BigQuery Dataset.

Projects with Linked Billing Account(s)

  • If Billing is enabled on the project and you have only one Cloud Billing account, the Billing Overview page will display.
  • If Billing is enabled on the project and you have MORE THAN one Cloud Billing account, a pop-up window will display: "Billing account Your Billing Account is linked to this project."

Once you have identified your billing configuration, continue to Create a BigQuery Dataset below.

Create a BigQuery Dataset​

Creating a BigQuery Dataset is only necessary if not following Create Vega Billing Service Account instructions. If following Create Vega Billing Service Account instructions then skip to Enable Cloud Billing Data Export to BigQuery

  1. Sign into the Google Cloud Console > Navigation menu > BigQuery
  2. In the Project drop down in the main Navigation bar, select the project you set up to contain your dataset. Note the Project ID, as you will need it in the next step. (Navigation: IAM & Admin > Settings > Project ID )
  3. In the Explorer panel, locate your project and click the vertical elipses menu, then select Create Dataset option.
  4. In Create dataset menu provide a Dataset ID and record, such as "VegaDataset".
  5. Click "CREATE DATASET". You will now see your Dataset entry under your project in the explorer menu.

Enable Cloud Billing Data Export to BigQuery​

  1. Sign into the Google Cloud Console.
  2. Open the console Navigation menu > Billing and verify that your desired Billing Account is selected.
  3. Under Cost Management select Billing export.
  4. Within the BigQuery export tab, under "Detailed usage cost" click "EDIT SETTINGS"
  5. For the Projects drop-down menu, select the project where you created the BigQuery Dataset previously.
  6. For the Dataset drop-down menu, select the Dataset you created in the Create a BigQuery Dataset section.
  7. Click "SAVE" You will see the report has been enabled. Note: You must enable BigQuery API for the project you set up to export data to BigQuery.

Billing Data Tables​

Once you enable Cloud Billing export to BigQuery, billing data tables are automatically created in the BigQuery dataset. Make note of the billing data table name. You will need this information during Billing Connector creation.

  1. Navigate to your deployment project within BigQuery.
  2. Click the dropdown next to your project to expose the VegaDataset created previously.
  3. Click the dropdown next to VegaDataset to expose the table name.
  4. Click the action menu next to the table, and select "Copy ID"
  5. Provide the Table ID in your clipboard to your Vega CSM or Onboarding Engineer. Example: testproj-123456.VegaDataset.gcp_billing_export_resource_v1_01C321_654321_321A1D

Step 3: Configure GCP Credentials​

Learn how to establish the identity of your Google Service Account to use it with the Vega Platform. To use a Service Account from the Vega Platform, you must first establish the identity of the Service Account. Public/private key pairs provide a secure way of accomplishing this goal. When you create a Service Account key, the public portion is stored on Google Cloud, while the private portion is only available to you.

Create Service Account and Service Account Key​

Note: if following Create Vega Billing Service Account then select the Service Account created previously and follow steps 8 through 11 below.

  1. In the Cloud Console, navigate to IAM & Admin > Service Accounts. Select your project from the drop-down at the top.
  2. Click "+ CREATE SERVICE ACCOUNT".
  3. In Service Account Details, enter "VegaServiceAccount" for Service Account Name.
  4. Enter "vegaserviceaccount" for Service account ID and a description such as "Vega Platform Data Retrieval Account".
  5. Click "CREATE AND CONTINUE".
  6. In the Role drop-down, select the role created previously (Vega Platform Integration role) and click "CONTINUE"
  7. Click "DONE".
  8. You should now see vegaserviceaccount@YOURPROJECT_ID_HERE.iam.gserviceaccount.com. Click "Actions > Manage Keys" in the same row.
  9. Click the ADD KEY drop down > Create new key.
  10. Leave JSON as the key type and click Create to create and download the JSON key file.
  11. Provide this file to your Vega CSM or Operations Engineer via a secure method or Encrypted email only. (Note: you will need Service account key creation enabled)

Service Account Key File​

  • The service account key file you downloaded is required for the Vega Platform to display your GCP cost detail.
  • The downloaded key has the following format, where private-key is the private portion of the public/private key pair:

JSON

{  
"type": "service_account",  
"project_id": "project-id",  
"private_key_id": "key-id",  
"private_key": "-----BEGIN PRIVATE KEY ---- \nprivate-key\n-----END PRIVATE KEY-----\n",  
"client_email": "service-account-email",  
"client_id": "client-id",  
"auth_uri": "https://accounts.google.com/o/oauth2/auth", 
"token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", 
"client_x509_cert_url":"https://www.googleapis.com/robot/ v1/metadata/x509/service-account-email" 
}

Step 4: Enable Cloud Resource Manager API for your GCP Project​

You will need to login to your Google Cloud Platform account and enable the Cloud Resource Manager API and link it to your GCP project.

Metadata for Google Cloud Platform resource containers is updated via the Cloud Resource Manager API.

  1. Login to your Google Cloud Platform (GCP) account.
  2. Navigate to the cloud resource manager API page in GCP and select the project you have created earlier to contain your BigQuery dataset. Click "ENABLE API".

Step 5: Assign Custom Role and Grant Permissions to Service Account​

In GCP, BigQuery uses a service account (also known as principal) to manage dataset permissions. As soon as you enable Cloud Billing export to BigQuery, GCP automatically adds a service account as an owner to the dataset that you specify in GCP.

The service account has the following format: gcp-role@your-project-234234.iam.gserviceaccount.com. By assigning roles to the service account, you grant the permissions associated with that role to the service account. Based on these permissions, the service account can fetch the required data from GCP to the Vega Platform.

Assign roles to the service account at organization level​

A service account can be assigned a role that has permissions at the org level. Then that service account will have permissions to all the project in the organization. In this example, we shall see how to assign a role created with minimum permissions to the service account (principal) at organization level. We shall use the role "Vega Platform Integration Role for GCP" created earlier. See Create Vega Admin Role in GCP with minimum permission sets

Assign role binding using the console:

  1. In the cloud console, go to IAM & Admin > IAM.
  2. Select the organization.
  3. Select the principal of your service account:
    1. To grant a role to a principal who already has other roles on the service account, find the row containing the principal’s email address, then click Edit principal in that row, then click add +Add another role.
    2. To grant a principal who does not already have other roles on the service account, click ADD from the top, then enter the principal's email address.
  4. In the Select a role box, browse down to Custom, and select the recently created role "Vega Platform Integration Role for GCP" which we created earlier. See Create Vega Admin Role in GCP with minimum permission sets.
  5. Click Save.

Assign role binding using cloud shell: (recommended)

  1. Identify your organization ID, and replace the string 123456789012 with your orgnization ID in the example command. (both references)

  2. Identify your service account email and replace the member reference in the example command. IAM & Admin > Service accounts > copy service account email button.

  3. Open your cloud shell in GCP and run the sample command with correct references to your organization, service account, and created role.

  4. Running the organization binding command will result in the output: "Updated IAM policy for organization [123456789012]" The output will reference your organization ID and list all bindings.

Assign roles to the service account at folder level (alternate method if you are not able to bind to the organization)

Use the procedure below to grant custom role(s) to the service account’s principal at folder level.

  1. In the cloud console, go to IAM & Admin > IAM.
  2. Select the folder.
  3. Select the principal of your service account:
    1. To grant a role to a principal who already has other roles on the service account, find the row containing the principal’s email address, then click Edit principal icon in that row, then click add +Add another role.
    2. To grant a principal who does not already have other roles on the service account, click ADD from the top, then enter the principal's email address.
  4. In the Select a role drop-down, browse to Custom, and select the recently created role Vega Platform Integration Role for GCP, Service Manager, and Service Advisor which we created earlier. See Create Vega Admin Role in GCP with minimum permission sets.
  5. Click Save. The principal is granted the roles on the service account.|

Step 6: Provide Information to Vega​

Vega will require the Service Account Name and generated Key for each of the Service Accounts created during onboarding in addition to the BigQuery Dataset Name and Table Name used for the Billing Export.

Adding GCP Projects to the Vega Platform​

Use the process below to add your Google Cloud Projects to the Vega Platform. Provider Accounts will use the Service Account named vegaserviceaccount that was created in Create Service Account and Service Account Key.

Do not use the Billing Service Account created in section Create Vega Billing Service Account as it has permissions limited to those necessary to extract data from BigQuery.

Step 1: Navigate to settings -> provider accounts and click 'Link Account, then select GCP'​

GCP Account Create

Step 2: Paste the service account JSON into the Service Account field, and optionally enter an alias​

  • Vega account alias: A friendly name for the account

If you aren't ready for your accounts to be ingested by Vega yet, you can toggle the 'Enabled' switch to 'Disabled' to prevent data ingestion.

NOTE: If you have multiple projects, repeat the above steps for each project or alternatively use the 'Bulk Import' feature to add multiple subscriptions at once.