Skip to main content

GCP Integration Guide

The following guide covers the steps to integrate your Google Cloud Platform (GCP) data with Vega Cloud.

Pre-Requisites: (enabled for the project you will use to deploy the resources below. Must have Billing enabled, and be linked to the same Cloud Billing account containing data for export to BigQuery.)

You will need to enable the following in your GCP environment:

  • Cloud Resource Manager API
    • Reads and updates metadata for resource containers.
  • Compute Engine API
    • Read VMs and volumes.
  • Cloud Billing API
    • Ingest billing data programmatically.
  • BigQuery API
    • Ingest BigQuery data.
  • Recommender API
    • Returns service recommendations for cost savings, security, and optimizations.

To enable these APIs:

  1. Navigate the Google Cloud Console to the library of APIs.
  2. On the Google Cloud Console home page, navigate to APIs & Services > Library.
  3. Search for the API you want to enable.
  4. In the API Library Search box, enter the name of the API you want to enable. Then press Enter to execute the search. Repeat these steps for each of:
  • Cloud Resource Manager API
  • Compute Engine API
  • Cloud Billing API
  • BigQuery API
  • Recommender API
  1. Enable the given API.
  2. In the list that appears, click the API name to navigate to that API page. If the API is not already enabled, click Enable.
  3. After you enable the given API, the console displays a details page for that API.
  4. Navigate to the console Home page.
  5. For each API you want to enable, navigate back to the home page and repeat these steps.

Step 1: Create Vega Custom Role in GCP with Minimum Permissions Sets​

Learn to create different roles as required with requisite permissions in GCP. Later those roles will be assigned to principals.

Vega Custom Role​

Creation of roles is very much dependent on your requirements. You can either create custom roles for various responsibilities such as monitoring, synchronizing, and then assign the required permissions to these roles. Or you can create a single custom role and assign all the permissions to that role. In our case, we create a single role. Later you can assign that role to the account whose service keys you will use to integrate with Vega Platform.

The Vega Platform uses billing data tables available within the BigQuery dataset to display the various cost reports. To perform the required actions, the role needs the permissions listed below.

Vega Provides a permission file containing all required access.

gcp-vega-admin-role.yml provides read only access to your Billing Account and any projects you assign the service account to.

Important: To create the Vega Admin custom role, you must have the iam.roles.create permission. By default, the owner of a project or an organization has this permission. Users who are not owners, including organization admins, must be assigned either the Organization Role Administrator role, or the IAM Role Administrator role.

Creation of custom role in GCP​

Create a custom role and assign permissions:

  1. Log in to your GCP platform.

  2. Click "Activate Cloud Shell" near the top right-hand corner. (You may need to approve any requests related to this session.)

  3. In another tab, browse to the github link provided above, and copy the yaml code using "Copy Raw File" button. Switch back to the GCP tab.

  4. With focus in Cloud Shell: open a new file by typing nano vega-permissions.yml then press "Enter" key. Nano editor will open a new file.

  5. Use Paste (Ctrl+V) function to place the copied code into the nano file buffer.

  6. Press the key combination (Ctrl+X) to enter buffer-save mode. Press "y" to display the new file name, and "Enter" again to save the file vega-permissions.yml. Nano will close.

  7. Configure and run the syntax from line 8 below if you are using an organization to manage GCP projects. If you are not using an organization, you will be required to deploy permissions at folder or project level. To do this skip to step 9.

  8. Replace ORGANIZATION_ID (recommended) in the following syntax to create the role within your organization: Organization Role Creation - gcloud iam roles create VegaPlatformRole --organization=ORGANIZATION_ID --file=vega-permissions.yml

  9. Replace PROJECT_ID (not recommended) in the following syntax to create the role in a single GCP project: Project Role Creation - gcloud iam roles create VegaPlatformRole --project=PROJECT_ID --file=vega-permissions.yml

    Note: The permissions must be added to the service account’s principle under all the projects linked with a billing account if not created at organization level.

Step 2: Enable Cloud Billing Data​

Required Permissions​

You must have the following permissions to enable and configure Google Cloud billing data export to a BigQuery dataset:

  • Billing account administrator role for the target Cloud billing account.
  • BigQuery User role for the Cloud project that contains the BigQuery dataset that will be used to store the Cloud Billing data.
  • Project Creator role on the organization or folder. To create a new project, you must have the following permissions:
    • resourcemanager.organizations.get
    • resourcemanager.projects.create

Enable Cloud Billing Data to BigQuery​

Select or create a project that will contain your dataset.

  1. In the Google Cloud console, select or create a Project to contain your BigQuery dataset. (Must have Billing enabled, and be linked to the same Cloud Billing account containing data for export to BigQuery.)
  2. With project selected in "Select a project" drop down at the top of the Google Cloud Console: Open the Navigation menu > Billing. Depending on your scenario, you will see one of the following:

Projects with No Linked Billing Account

  1. Enable Billing on the project by either selecting Link a billing account or Manage billing accounts options.
  2. Continue to Create a BigQuery Dataset.

Projects with Linked Billing Account(s)

  • If Billing is enabled on the project and you have only one Cloud Billing account, the Billing Overview page will display.
  • If Billing is enabled on the project and you have MORE THAN one Cloud Billing account, a pop-up window will display: "Billing account Your Billing Account is linked to this project."

Once you have identified your billing configuration, continue to Create a BigQuery Dataset below.

Create a BigQuery Dataset​

  1. Sign into the Google Cloud Console > Navigation menu > BigQuery
  2. In the Project drop down in the main Navigation bar, select the project you set up to contain your dataset. Note the Project ID, as you will need it in the next step. (Navigation: IAM & Admin > Settings > Project ID )
  3. In the Explorer panel, locate your project and click the vertical elipses menu, then select Create Dataset option.
  4. In Create dataset menu provide a Dataset ID and record, such as "VegaDataset".
  5. Click "CREATE DATASET". You will now see your Dataset entry under your project in the explorer menu.

Enable Cloud Billing Data Export to BigQuery​

  1. Sign into the Google Cloud Console.
  2. Open the console Navigation menu > Billing and verify that your desired Billing Account is selected.
  3. Under Cost Management select Billing export.
  4. Within the BigQuery export tab, under "Detailed usage cost" click "EDIT SETTINGS"
  5. For the Projects drop-down menu, select the project where you created the BigQuery Dataset previously.
  6. For the Dataset drop-down menu, select the Dataset you created in the Create a BigQuery Dataset section.
  7. Click "SAVE" You will see the report has been enabled. Note: You must enable BigQuery API for the project you set up to export data to BigQuery.

Billing Data Tables​

Once you enable Cloud Billing export to BigQuery, billing data tables are automatically created in the BigQuery dataset. Make note of the billing data table name. You will need this information during Billing Connector creation.

  1. Navigate to your deployment project within BigQuery.
  2. Click the dropdown next to your project to expose the VegaDataset created previously.
  3. Click the dropdown next to VegaDataset to expose the table name.
  4. Click the action menu next to the table, and select "Copy ID"
  5. Provide the Table ID in your clipboard to your Vega CSP. Example: testproj-123456.VegaDataset.gcp_billing_export_resource_v1_01C321_654321_321A1D

Step 3: Configure GCP Credentials​

Learn how to establish the identity of your Google Service Account to use it with the Vega Platform. To use a Service Account from the Vega Platform, you must first establish the identity of the Service Account. Public/private key pairs provide a secure way of accomplishing this goal. When you create a Service Account key, the public portion is stored on Google Cloud, while the private portion is only available to you.

Create Service Account and Service Account Key​

  1. In the Cloud Console, navigate to IAM & Admin > Service Accounts. Select your project from the drop-down at the top.
  2. Click "+ CREATE SERVICE ACCOUNT".
  3. In Service Account Details, enter "VegaServiceAccount" for Service Account Name.
  4. Enter "vegaserviceaccount" for Service account ID and a description such as "Vega Platform Data Retrieval Account".
  5. Click "CREATE AND CONTINUE".
  6. In the Role drop-down, select the role created previously (Vega Platform Integration role) and click "CONTINUE"
  7. Click "DONE".
  8. You should now see vegaserviceaccount@YOURPROJECT_ID_HERE.iam.gserviceaccount.com. Click "Actions > Manage Keys" in the same row.
  9. Click the ADD KEY drop down > Create new key.
  10. Leave JSON as the key type and click Create to create and download the JSON key file.
  11. Provide this file to your Vega CSM via a secure method or Encrypted email only. (Note: you will need Service account key creation enabled)

Service Account Key File​

  • The service account key file you downloaded is required for the Vega Platform to display your GCP cost detail.
  • The downloaded key has the following format, where private-key is the private portion of the public/private key pair:

JSON

{  
"type": "service_account",  
"project_id": "project-id",  
"private_key_id": "key-id",  
"private_key": "-----BEGIN PRIVATE KEY ---- \nprivate-key\n-----END PRIVATE KEY-----\n",  
"client_email": "service-account-email",  
"client_id": "client-id",  
"auth_uri": "https://accounts.google.com/o/oauth2/auth", 
"token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", 
"client_x509_cert_url":"https://www.googleapis.com/robot/ v1/metadata/x509/service-account-email" 
}

Step 4: Enable Cloud Resource Manager API for your GCP Project​

You will need to login to your Google Cloud Platform account and enable the Cloud Resource Manager API and link it to your GCP project.

Metadata for Google Cloud Platform resource containers is updated via the Cloud Resource Manager API.

  1. Login to your Google Cloud Platform (GCP) account.
  2. Navigate to the cloud resource manager API page in GCP and select the project you have created earlier to contain your BigQuery dataset. Click "ENABLE API".

Step 5: Assign Custom Role and Grant Permissions to Service Account​

In GCP, BigQuery uses a service account (also known as principal) to manage dataset permissions. As soon as you enable Cloud Billing export to BigQuery, GCP automatically adds a service account as an owner to the dataset that you specify in GCP.

The service account has the following format: gcp-role@your-project-234234.iam.gserviceaccount.com. By assigning roles to the service account, you grant the permissions associated with that role to the service account. Based on these permissions, the service account can fetch the required data from GCP to the Vega Platform.

Assign roles to the service account at organization level​

A service account can be assigned a role that has permissions at the org level. Then that service account will have permissions to all the project in the organization. In this example, we shall see how to assign a role created with minimum permissions to the service account (principal) at organization level. We shall use the role "Vega Platform Integration Role for GCP" created earlier. See Create Vega Admin Role in GCP with minimum permission sets

Assign role binding using the console:

  1. In the cloud console, go to IAM & Admin > IAM.
  2. Select the organization.
  3. Select the principal of your service account:
    1. To grant a role to a principal who already has other roles on the service account, find the row containing the principal’s email address, then click Edit principal in that row, then click add +Add another role.
    2. To grant a principal who does not already have other roles on the service account, click ADD from the top, then enter the principal's email address.
  4. In the Select a role box, browse down to Custom, and select the recently created role "Vega Platform Integration Role for GCP" which we created earlier. See Create Vega Admin Role in GCP with minimum permission sets.
  5. Click Save.

Assign role binding using cloud shell: (recommended)

  1. Identify your organization ID, and replace the string 123456789012 with your orgnization ID in the example command. (both references)

  2. Identify your service account email and replace the member reference in the example command. IAM & Admin > Service accounts > copy service account email button.

  3. Open your cloud shell in GCP and run the sample command with correct references to your organization, service account, and created role.

  4. Running the organization binding command will result in the output: "Updated IAM policy for organization [123456789012]" The output will reference your organization ID and list all bindings.

Assign roles to the service account at folder level (alternate method if you are not able to bind to the organization)

Use the procedure below to grant custom role(s) to the service account’s principal at folder level.

  1. In the cloud console, go to IAM & Admin > IAM.
  2. Select the folder.
  3. Select the principal of your service account:
    1. To grant a role to a principal who already has other roles on the service account, find the row containing the principal’s email address, then click Edit principal icon in that row, then click add +Add another role.
    2. To grant a principal who does not already have other roles on the service account, click ADD from the top, then enter the principal's email address.
  4. In the Select a role drop-down, browse to Custom, and select the recently created role Vega Platform Integration Role for GCP, Service Manager, and Service Advisor which we created earlier. See Create Vega Admin Role in GCP with minimum permission sets.
  5. Click Save. The principal is granted the roles on the service account.|

Step 6: Grant GCP Billing Bucket Permissions to Service Account​

To enable Billing Data to be retrieved from GCP by the Vega Platform, additional manual steps must occur. Your Vega Client Success Manager will further instruct you as necessary.

Adding GCP Projects to the Vega Platform​

Step 1: Navigate to settings -> provider accounts and click 'Link Account, then select GCP'​

GCP Account Create

Step 2: Paste the service account JSON into the Service Account field, and optionally enter an alias​

  • Vega account alias: A friendly name for the account

If you aren't ready for your accounts to be ingested by Vega yet, you can toggle the 'Enabled' switch to 'Disabled' to prevent data ingestion.

NOTE: If you have multiple projects, repeat the above steps for each project or alternatively use the 'Bulk Import' feature to add multiple subscriptions at once.