HashiCorp's Terraform Enterprise is a HashiCorp supported multi-cloud cloud provisioning and management system. There are two flavors in this

1. Terraform Cloud - It is SaaS and hosted by Hashicorp for you.
2. Terraform Enterprise (TFE) - Hosted on premise by the customer

The API is consistent across Terraform Cloud and Enterprise. In this article when i mean Terraform Enterprise, Terraform Cloud is included as well.

This article covers how Cloud Management in ServiceNow integrates with TFE.

Cloud Management support for Cloud-Native and Multi-Cloud Templating Systems

In Cloud Management, we want to make sure that our customers have a consistent way of dealing with various templating systems. This includes the specifying the templates, creating/generating catalog items from the templates and consuming the catalog items.

The below diagram explains it:

We want and have a consistent way to ingest any template and use appropriate and consistent execution mechanism for provisioning/managing any template. This implies there will be consistent treatment for CFT, ARM, Terraform opensource, Terraform Enterprise, GDM and any other supported template. The consistent treatment implies that discovery, catalog item creation, template ingestion, catalog item consumption and the provisioned stack lifecycle are all consistent across all these clouds.

The overall integration flow with TFE would be like this:

The Admin persona would setup the Terraform enterprise CI record and associate the credentials, URL. She would then discover the TFE related resources like VCS, Repo, workspaces etc.

The Designer persona would then create a catalog item and point to the right Repo within an appropriate VCS. ServiceNow will auto-populate the catalog item based on the variables et al. The auto-population would take into consideration any metadata info provided. The designer then does any addition decoration like icons etc. She would then mark the catalog item as available for the customer.

The End-user persona would then get to the portal and choose the catalog item. He will fill in the values based on the choices the designer has made. He will then submit it. He can use API to do the same. The system would then execute the backing terraform template and invoke the TFE API to create workspace and apply it as well. The system would create a stack representing the resources that got provisioned in this call. The CMDB would also get populated properly.

Setup and Discovery - Admin Persona

This is the first part of the TFE integration. You would need to

1. Create a credential
2. Then use that credential to create a TFE provider organization record in Cloud Management
3. Then discover the TFE organization

Create Credential

We need to create the credential record for the TFE API keys. You would need to get to the Cloud Admin Portal and get to Credentials section.

You would need to create the credential record. For this you need to choose then need to specify the 'API Key Credentials'.

Then create the credentials by providing the API keys from TFE for the Terraform Organization. Provide a meaningful name and the API Key in the creation screen. The API key would be available from the Terraform Enterprise organization setting screen. The credential creation screen will look like this:

TFE Credential key (API Key) must start with Bearer. (For example something like this ==> Bearer mabcdefgh…)

ThenClick submit.

After you create the credential, make sure that you create an alias for the credential as well. The alias should be of type 'Credential'.

Click on the 'lock' icon next to the credential alias.

Click Submit.

Choose the credential alias you just created.

Click update.

This alias association is critical as the underlying IntegrationHub calls depend on the credential alias.

Create TFE Organization Record

You would need to get to the Cloud Admin Portal and get to Config Management section.

Click on the 'New' button to create a new TFE provider record. This will open the create popup screen like this:

Provide a name unique to this TFE.

Choose the Provider as 'Terraform Enterprise'.

Provide the org name that corresponds to the TFE organization.

For Terraform Cloud, provide 'https://app.terraform.io/api/v2' as the URL. For Terraform Enterprise get the appropriate URL from your terraform administrator.

Choose the Server Type as either 'Cloud' or 'Enterprise'.

Choose the credential that corresponds to this TFE instance.

Then save it.

Discover TFE Organization

Post creation of the TFE organization, it would show up in the landing page. Click on the TFE organization record you just created and it would show up like this.

You would see the resource types which we discover in a TFE organization. It would be empty in the beginning. Now click on the 'Discover Now' button to start the discovery of this organization. The discovery would take a few moments to complete. Then you would see something like this:

Click on the 'Tfe VCS' to get the list of VCS providers associated to this organization.

Any repo under any of these VCS providers can be used by Cloud Management. For this we will need credential information about the VCS system so that we can read the Terraform files for the ingestion process which will come later. This is something which is very critical for catalog item creation. You would need to create the Github credential record for the VCS system(s). As usual you create the API key credential record and associate it to the VCS records.

Create a API Key Credential by going to the credentials menu. The click New. Then choose an API Key Credential.

Key for VCS system should be like this token< GENERATED-KEY >. (token fabcdefgh123.....…)

Click submit.

Click on the Terraform Enterprise VCS record. Make sure that the proper credential record is associated.

Click on the 'Find Branches and Repositories'. This will discover all the repos under the the VCS system at a high level. This is so that the catalog item designer can choose to expose any of the repo as a catalog item later. You would do this to each of the VCS system that you want to.

Catalog Item Creation - Designer Persona

One of the most important value points of TFE integration with Cloud Management is the ease with which one can expose a terraform configuration as a full fledged catalog item for the end user to consume. And with the catalog item comes all the goodness of ServiceNow with respect to Governance and CMDB support etc.

Get to the Cloud Admin Portal and from there to the Cloud Catalog Item in the Design section.

Click on 'New' button to create a new catalog item.

Choose the source as 'Configuration Management Template' and Provider Type as 'Terraform Enterprise'.

Then choose your specific TFE provider as the provider.

Click save to save the record. Then you would see a screen like this.

So far we have just created the catalog item and it is almost an empty record. Now we will have to specify the particular github repo so that this catalog item can front end it. Click on the 'New' button in the 'Cloud Template' tab in the lower part of the screen.

The 'Configuration Installable' will show the list of repos which are pertinent to the VCS systems that are associated with this Terraform organization. The 'Configuration Installable' is nothing but the provisionable terraform templates in this context.

Click on the search icon next to the Configuration Installable. A screen to pick the github repo will show up and will look like this:

Choose the appropriate repo and the version. Then click submit. On submission, the system takes in the github repo info, uses the credential info from the associated VCS system and reads the terraform files. It gets the info about the provider, variables etc and then populates the template version parameters.

Click the 'Activate' button. This will take in the template version parameters and other associated information and populate the catalog item appropriately. It will create the mgmt variables as well as the terraform related variables and create any catalog client scripts etc as well. It will make the catalog item as a completely working one.

Then check the 'Active' flag on the catalog item and save. Now your catalog item is ready for end user consumption.

Consumption - End User Persona

Provisioning

The end user can get to the Cloud User portal to order this catalog item (offering).

Fill in information and click 'Next'.

Fill-in/choose the appropriate info and click submit. The system will then communicate with Terraform Enterprise, create the workspace, apply it. When the apply is complete, the system will do a pointed discovery of the provisioned resources and populate the CMDB. In addition it will create the stack and associate these CIs to the stack as well.

Click on the 'View stack details' to get the details of the stack that was provisioned.

The stack info would show up like this:

Deprovisioning

A provisioned stack can be deprovisioned later by the user. She can choose the stack and then click on the 'Deprovision' operation and click submit. The system will then communicate with theTerraform Enterprise system and decommission the workspace.

In addition to deprovisioning the stack/workspace, CMP provides you the ability to do Day-2 operations directly on the resource contained within the stack itself. For these operations, it would use the provider's API to do the needful.

Additional Info

In this article the TFE integration with CMP was described. We went over the various steps that needs to be done to make it happen. In coming days we will be adding articles that explain how to use the 'Metadata' snippets to enrich the terraform templates and how it reduces the TCO. We will also go over the additional advantages CMP provides on top of the TFE integration.