GCP Integration Guide
The following guide covers the steps to integrate your Google Cloud Platform (GCP) data with Vega Cloud using GCP Deployment Manager to manage the resources.
Pre-Requisites
In order to create the necessary resources to integrate with the Vega Platform you will need:
A GCP Project, preferably within an Organization
The Project that you are deploying to must have Billing Enabled and be linked to the same Cloud Billing Account containing data to export to BigQuery
A GCP Project with the GCP IAM Role
roles/owner
granted to the user performing the Integration- GCP Deployment Manager is unable to manage IAM Roles and Bindings out of the box and requires temporarily elevated permissions to perform these actions, additional information can be found in the GCP Deployment Manager documentation.
To enable GCP Billing Exports to the BigQuery Dataset, the user configuring the Vega Integration requires:
- One of the following GCP Roles for the Cloud Billing Account to configure exports for:
- Billing Account Administrator (
roles/billing.admin
) - Billing Account Costs Manager (
roles/billing.costsManager
)
- Billing Account Administrator (
- BigQuery User GCP Role (
roles/bigquery.user
) in the Project being used to house the Billing Export
- One of the following GCP Roles for the Cloud Billing Account to configure exports for:
The ability to create GCP Roles and Bind Members at the Organization or Project level depending on your needs, choose a path below.
Organization level GCP Role (recommended)
To create a GCP Role at the Organization level (recommended) then the following is required:
GCP IAM Role
roles/resourcemanager.organizationAdmin
- this is used to temporarily elevate permissions of the GCP API Service Account that exists by default in all Projects
Project level GCP Role
To create a GCP Role at the Project level (not recommended) then the following is required:
roles/iam.roleAdmin
at the Project we are using for the Vega Integration
Certain Google Cloud Service APIs need to be enabled (see below)
Note: granting additional permissions to the GCP API Service Account does not grant Vega additional permissions. It allows Deployment Manager to manage IAM Resources on your behalf. These Roles may be revoked once the resources have been deployed.
Google Cloud Service APIs
The below Google Cloud Service APIs need to be enabled for the Google Cloud Project that will contain the BigQuery Billing Dataset.
You will need to enable the following Google Cloud Service APIs:
- Cloud Resource Manager API (cloudresourcemanager.googleapis.com)
- Reads and updates metadata for resource containers.
- Compute Engine API (compute.googleapis.com)
- Read VMs and volumes.
- Cloud Billing API (cloudbilling.googleapis.com)
- Ingest billing data programmatically.
- BigQuery API (bigquery.googleapis.com)
- Ingest BigQuery data.
- Recommender API (recommender.googleapis.com)
- Returns service recommendations for cost savings, security, and optimizations.
- Cloud Storage API (storage.googleapis.com)
- Allows interactions with Cloud Storage Buckets.
- Deployment Manager V2 API (deploymentmanager.googleapis.com)
- Deploy resources from a configuration.
- Identity and Access Management API (iam.googleapis.com)
- Manages identity and access control for GCP resources.
To enable these APIs choose a method below choose one of the two methods below.
Google Cloud CLI (recommended)
- Execute this command to enable the necessary Service APIs
gcloud services enable cloudresourcemanager.googleapis.com compute.googleapis.com cloudbilling.googleapis.com bigquery.googleapis.com recommender.googleapis.com storage.googleapis.com deploymentmanager.googleapis.com iam.googleapis.com
Google Cloud User Interface (web browser)
- Navigate to the Google Cloud Console to the library of APIs.
- On the Google Cloud Console home page, navigate to APIs & Services > Library.
- Search for the API you want to enable.
- In the API Library Search box, enter the name of the API you want to enable. Then press Enter to execute the search.
- Enable the given API.
- In the list that appears, click the API name to navigate to that API page. If the API is not already enabled, click Enable.
- After you enable the given API, the console displays a details page for that API.
- Repeat this step for each of the Service APIs listed above.
- Navigate to the console Home page.
- For each API you want to enable, navigate back to the home page and repeat these steps.
Deployment Manager
Vega currently supports GCP Roles at two levels of the organizational hierarchy - Organizations and Projects. The recommended method is to deploy GCP Roles to the Organization to avoid needing to recreate the GCP Role and Service Accounts for each Project you wish to add to the Vega Platform.
Vega Products and the Resources Deployed
Templates
The Vega GCP Deployment Manager Resource Templates are comprised of Jinja files, you only need the files that are relevant to your environment and deployment.
File Name | Description | Link |
---|---|---|
v-inform.jinja | This is the base resource template to be deployed, it enables read access to your Project(s) to collect resource metrics and generate recommendations for your environment. | link |
billing-export.jinja | This is another base resource template to be deployed, it enables Vega to collect your usage data within GCP. This enables financial and usage visualizations along with amortization of costs | link |
v-operate.jinja | This template should only be included if you are using Vega's Enterprise Parking functionality | link |
Resources
Each of these Deployment Manager Templates deploys resources to your Project as described below.
Source Template | Resource Name | Resource ID | Description | Notes |
---|---|---|---|---|
v-inform.jinja | vega-service-account | vega-sa | The Service Account Vega will use to connect to your environment. | |
v-inform.jinja | vega-inform-custom-role | vegaInformRole | A Custom GCP Role that provides read only access to GCP Resource Metadata. | |
v-inform.jinja | vega-inform-custom-role-binding | Binds vega-inform-custom-role to the Service Account created at vega-service-account . | ||
billing-export.jinja | vega-billing-sa-bq-jobuser-binding | Binds the GCP Role roles/bigquery.jobUser (link) to vega-service-account at the Project level. | This GCP Role provides Vega the ability to create BigQuery Jobs within the Project, but does not provide access to data. | |
billing-export.jinja | vega-billing-dataset | VegaBillingDataset | Creates a BigQuery Dataset in the Project and binds the GCP Role roles/bigquery.dataViewer (link) to vega-service-account , limiting access to this BigQuery Dataset. | Billing Export will be configured to load data into this BigQuery Dataset. |
billing-export.jinja | vega-billing-export-{{ PROJECT_NUMBER }} | vega-billing-export-{{ PROJECT_NUMBER }} | A GCP Cloud Storage Bucket that will be used to export Billing Data from the BigQuery Dataset. Additionally binds two roles, both scoped to the Bucket: roles/storage.objectUser (link) to vega-service-account and roles/storage.admin (link) to the GCP API Service Account (allows Deployment Manager to manage this resource). | The template includes a Lifecycle Policy on the Bucket to delete objects after 30 days. |
v-operate.jinja | vega-operate-actions-role | vegaOperateRole | A Custom GCP Role that provides the permissions necessary for Vega's Enterprise Parking | |
v-operate.jinja | vega-operate-custom-role-binding | Binds vega-operate-actions-role to the Service Account created at vega-service-account |
Either download the Template files or reference the files by their Github Raw URL in your Deployment Manager Resource Configuration when creating your configuration, which we will walk through below.
Deployment
Step 1: Gather Environment Info
First we need to gather some information about the environment in order to execute CLI commands to prepare the environment for deployment.
If you do not know what the Project ID is (see screenshot below) then log in to your Google Cloud account and select the Project that we will be configuring for use with Vega. Copy the
Project ID
and theProject number
to a text editor or another location you will be able to reference.If you are deploying the GCP Role to the Organization (recommended) then execute this command, substituting in PROJECT_ID.
gcloud projects get-ancestors <PROJECT_ID>
For example with output:
gcp_user@cloudshell:~$ gcloud projects get-ancestors circular-hybrid-431319-d7
ID: circular-hybrid-431319-d7
TYPE: project
ID: 2###########
TYPE: folder
ID: 8##########
TYPE: organizationCopy the Organization ID from the output to the same location you copied the Project ID and Project Number.
Step 2: Assign GCP API Service Account Role
Execute the below commands to bind the roles/owner
GCP Role to the GCP API Service Account. Substitute in values for
PROJECT_ID
and PROJECT_NUMBER
.
Bind
roles/owner
at the Project level to the Project's GCP API Service Accountgcloud projects add-iam-policy-binding <PROJECT_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/ownerIf deploying Roles to the GCP Organization then bind both
roles/iam.organizationRoleAdmin
androles/resourcemanager.organizationAdmin
to the Project's GCP API Service Account, substituting in theORGANIZATION_ID
andPROJECT_NUMBER
as necessary.gcloud organizations add-iam-policy-binding <ORGANIZATION_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/resourcemanager.organizationAdmin
gcloud organizations add-iam-policy-binding <ORGANIZATION_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/iam.organizationRoleAdmin
Note: if a binding condition is required (organization bindings frequently require one) it will be indicated in the console with a list of selections - choose
None
by entering the corresponding number next to that condition and pressingEnter
.
Step 3: Deployment Manager Resource Configuration
While you are able to manage GCP Deployment Manager Configurations using the command line we do not recommend doing so. We prefer to create a GCP Deployment Manager Configuration that defines our templates and the resources we will be creating. Configuration files (and templates) can be included in Infrastructure as Code solutions and will allow for repeatable deployments and simpler paths to updates in the future.
There are a number of GCP Deployment Manager Resource Configuration samples in that can be found in the Vega Cloud customer-samples GCP directory. These can be referenced to build the configuration appropriate to your use-case.
To create your GCP Deployment Manager Configuration, create a yaml
file in the directory you are using for this process,
give the file a name that is meaningful to you and your organization.
A basic sample configuration is available below that is sufficient for the vInform Vega Product.
Your exact configuration may vary depending on your use-case and any GCP Organization Policies that may be in place. If you have any questions reach out to your Customer Success Manager or Onboarding Team.
- Create a
yaml
file wherever you are executinggcloud
cli commands from, the name of the file should be meaningful to you, e.g.vega-integration.yaml
- Populate the file with a configuration appropriate to your use-case
- The sample configuration below is an example of the typical customer integration
Note: If you are deploying the GCP Role at the Organization level then fill in the
resources[0].properties.OrganizationId
. If you are deploying at the Project level then set it to an empty string, e.g.resources[0].properties.OrganizationId: ""
Sample Configuration
The below sample configuration has been copied from here with small modifications to Store Templates on an External Host.
To summarize: when templates are stored at a static url we do not need to import
the templates, but instead we reference
the template in the resource's type
key as seen below.
resources:
- name: vInform
type: https://raw.githubusercontent.com/vegacloud/customer-samples/main/latest/gcp/templates/v-inform.jinja
properties:
OrganizationId: 8##########
- name: Vega-Billing-Export
type: https://raw.githubusercontent.com/vegacloud/customer-samples/main/latest/gcp/templates/billing-export.jinja
properties:
BindingType: $(ref.vInform.BindingType)
VegaServiceAccountEmail: $(ref.vega-service-account.email)
metadata:
dependsOn:
- vInform
A typical Resource Configuration for a Customer who is using the vInform offering from Vega is above. Using the tables at Vega Products and the Resources Deployed we expect this configuration to:
- deploy the base Vega template
v-inform.jinja
, which will create:- a Service Account named
vega-service-account
- a GCP Custom IAM Role at the Organization level named
vegaInformRole
- a binding of the Organization Role
vegaInformRole
to the Service Accountvega-service-account
- a Service Account named
- deploy the Billing Export template
billing-export.jinja
, which will create:- a binding of the GCP defined Role
roles/bigquery.jobUser
to the Service Accountvega-service-account
- a BigQuery Dataset named
VegaBillingDataset
- a GCP Cloud Storage Bucket named
vega-billing-export-<PROJECT_NUMBER>
- a binding of the GCP defined Role
Step 4: Deploy the Resource Configuration
Substitute into the command below DEPLOYMENT_NAME
(lower-case only) and RESOURCE_CONFIG_YAML
(the yaml created in the previous step).
In the same directory as your Deployment Manager Resource Configuration, execute the following gcloud CLI command:
gcloud deployment-manager deployments create <DEPLOYMENT_NAME> --config <RESOURCE_CONFIG_YAML>
Step 5: Cloud Billing Export
Enable Cloud Billing Export to BigQuery
Sign into the Google Cloud Console (if not already signed in).
Open the Console's Navigation menu > Billing and verify that your desired Billing Account is selected.
Under Cost Management select Billing export.
Depending on your scenario, you will see one of the following:
Projects with No Linked Billing Account
Enable Billing on the Project by either selecting Link a Billing Account or Manage Billing Account Options.
Projects with Linked Billing Account(s)
a. If Billing is enabled on the Project and you have only one Cloud Billing Account then the Billing Overview Page will display.
b. If Billing is enabled on the Project and you have MORE THAN one Cloud Billing Account a pop-up window will display: "Billing Account <YOUR_BILLING_ACCOUNT> is linked to this Project."
Within the BigQuery export tab, under "Detailed usage cost" click "EDIT SETTINGS"
For the Projects drop-down menu, select the project where you created the BigQuery Dataset previously.
For the Dataset drop-down menu, select the Dataset you created in the Create a BigQuery Dataset section.
Click "SAVE" You will see the report has been enabled. Note: You must enable BigQuery API for the project you set up to export data to BigQuery.
Billing Data Tables
Once you enable Cloud Billing export to BigQuery, billing data tables are automatically created in the BigQuery Dataset. Make note of the billing data table name. You will need this information during Billing Connector creation.
- Navigate to your deployment project within BigQuery.
- Click the dropdown next to your project to expose the VegaDataset created previously.
- Click the dropdown next to VegaDataset to expose the table name.
- Click the action menu next to the table, and select "Copy ID"
- Provide the Table ID in your clipboard to your Vega CSM or Onboarding Engineer. Example:
testproj-123456.VegaBillingDataset.gcp_billing_export_resource_v1_01C321_654321_321A1D
Step 6: Create Service Account Credentials
Now we need to create a Google Credentials File so Vega can authenticate to the Service Account created previously.
This could be included with the Deployment Manager Resource Templates but we prefer not to as this would include the Service Account credentials in the output in plain text.
How you perform this task is up to you and may vary depending on your environment. Just ensure you send the Google Credentials File to your Customer Success Manager or Onboarding Team.
This author prefers to do this through the CLI, though it can be accomplished through the Google Cloud Console as well.
gcloud CLI
Substitute SERVICE_ACCOUNT_EMAIL
for the Vega Service Account's email, which by default is vega-sa@<PROJECT_ID>.iam.gserviceaccount.com
gcloud iam service-accounts keys create ~/vega-sa.json --iam-account <SERVICE_ACCOUNT_EMAIL>
Please provide the resulting file, vega-sa.json
in the above example, to your Vega Customer Success Manager or Onboarding Team.
Google Cloud Console (web UI)
- In the Cloud Console navigate to IAM & Admin > Service Accounts, select the Project from the drop-down menu at the top.
- Find the Service Account created by Deployment Manager, default email is
vega-sa@<PROJECT_ID>.iam.gserviceaccount.com
with the nameVega Inform Service Account
- On the right side of the page in the Actions column, click the ellipsis and select Manage Keys
- Click the ADD KEY drop down and select Create new key
- Leave JSON as the key type and click CREATE, this will cause the key to be generated and the file downloaded.
- Provide the downloaded JSON Google Credentials File to your Customer Success Manager or Onboarding Team.
The resulting file will look something like this:
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY ---- \nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url":"https://www.googleapis.com/robot/ v1/metadata/x509/service-account-email"
}
Step 7: Provide Information to Vega
Vega will require the Service Account Name and generated Key for each of the Service Accounts created during onboarding in addition to the BigQuery Dataset Name and Table Name used for the Billing Export.
Step 8: Remove Elevated Permissions (optional)
If you do not want the GCP API Service Account to retain the GCP Roles we bound earlier then those bindings may be removed, however it is important to note that Deployment Manager will not have the ability to update or delete these Roles or Bindings without re-binding the Roles to the GCP API Service Account.
Which of the below commands you execute is dependent on your environment, configuration, organizational requirements, and the commands you executed previously as part of this setup.
gcloud organizations remove-iam-policy-binding <ORGANIZATION_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/resourcemanager.organizationAdmin
gcloud organizations remove-iam-policy-binding <ORGANIZATION_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/iam.organizationRoleAdmin
gcloud projects remove-iam-policy-binding <PROJECT_ID> \
--member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
--role roles/owner
Adding GCP Projects to the Vega Platform
Use the process below to add your Google Cloud Projects to the Vega Platform. Provider Accounts will use the Service Account named vega-sa
that was created by GCP Deployment Manager in Step 3.
Step 1: Navigate to settings -> provider accounts and click 'Link Account, then select GCP'
Step 2: Paste the service account JSON into the Service Account field, and optionally enter an alias
- Vega account alias: A friendly name for the account
Step 3: Click 'Link Account' to complete the process
If you aren't ready for your accounts to be ingested by Vega yet, you can toggle the 'Enabled' switch to 'Disabled' to prevent data ingestion.
NOTE: If you have multiple projects, repeat the above steps for each project or alternatively use the 'Bulk Import' feature to add multiple subscriptions at once.