How to Use AWS Service Catalog with HashiCorp Terraform Cloud

By Emel De Jesus Mendoza Patino, Solutions Architect – AWS
By Welly Siauw, Principal Partner Solutions Architect – AWS
By Taylor Brooks, Software Engineer – HashiCorp

HashiCorp

Customers use AWS Service Catalog to create and manage a catalog of IT services and products approved for use on Amazon Web Services (AWS). These products include resources such as virtual machines, databases, or complete artificial intelligence (AI) architectures curated by the AWS Service Catalog administrator.

End users such as data scientists or AI specialists can provision the approved IT services in self-service mode. AWS Service Catalog also integrates with Terraform Cloud, a HashiCorp managed service offering. By using AWS Service Catalog with Terraform Cloud, customers can benefit from a self-service provisioning model and remove the heavy lifting of managing their Terraform infrastructure.

In this post, we will show how to use AWS Service Catalog Engine for Terraform Cloud to provision your products. HashiCorp is an AWS Specialization Partner and AWS Marketplace Seller that provides consistent workflows to provision, secure, connect, and run any infrastructure for any application.

AWS Service Catalog Engine for Terraform Cloud

AWS Service Catalog supports infrastructure as code (IaC) tools including AWS CloudFormation and HashiCorp Terraform. AWS Service Catalog administrators import the IaC template/configuration file as a Service Catalog product. Collections of products are grouped into Service Catalog portfolios, which are then shared and distributed to end users, which are teams within your organization that deploy resources in their AWS accounts.

Each product type has a specific provisioning engine to deploy the resources. Before using the Terraform Cloud product type, the Terraform Cloud Reference Engine (TFC-RE) must be installed in the customer AWS account. The TFC-RE can be deployed in the hub-and-spoke model, allowing Service Catalog administrators to install TFC-RE in a central account (“hub”) and share the portfolio with multiple AWS accounts (“spoke”).

Let’s review the workflow for a new product provisioning in AWS Service Catalog using Terraform Cloud. First, a Service Catalog end user provisions a Terraform Cloud product type from the spoke account. In this example, let’s assume the product is an Amazon SageMaker notebook instance.

AWS Service Catalog sends the request to the TFC-RE in the hub account via Amazon Simple Queue Services (SQS). The request contains information such as the location of the Terraform configuration file and the Service Catalog launch role constraints. The request is processed using AWS Lambda and sent to Terraform Cloud to perform Terraform workflow (plan, apply or destroy).

Terraform Cloud authenticates to the spoke account using the launch role and provisions the product. Once the provisioning is completed, TFC-RE sends the event completion to AWS Service Catalog. The entire workflow is orchestrated using AWS Step Functions and can scale according to customer requirements.

Figure 1 – Terraform Cloud with AWS Service Catalog.

Deploying Terraform Cloud Reference Engine

To start using AWS Service Catalog with Terraform Cloud, you need to install the TFC-RE that conﬁgures the infrastructure required for the Terraform Cloud Engine to work with AWS Service Catalog.

Let’s review the prerequisites to deploy the TFC-RE:

Basic understanding of Terraform and HashiCorp Configuration Language.
Designated AWS account that acts as the Service Catalog hub. The TFC-RE is deployed in this hub account and serves the Terraform provisioning for all spoke accounts. This account also acts as the owner of the portfolio and products.
Terraform Cloud subscription with a Standard billing plan. This is required in order to use the Team Management feature. You can sign up for Terraform Cloud free trial.
Create a Terraform Cloud API user-token with the permissions for team creation. Refer to the documentation for more details about users/tokens in Terraform Cloud.
Create a Terraform Cloud workspace and make sure to name the workspace: tfc_re_bootstrap. Select the CLI-driven workflow workspace and set the workspace execution mode to Local. Use this documentation for instruction to create a workspace.

Now, with the prerequisites completed, let’s deploy the TFC-RE:

Log in to your AWS Service Catalog hub account and navigate to the AWS Secrets Manager console.
To create a new secret, select Store a new secret and choose Other type of secret.
Select Plaintext as the format and enter the Terraform Cloud API token as the secret value.
You can select the default aws/secretsmanager encryption key, or use your own customer managed key.
Give the secret name: terraform-cloud-credentials-for-bootstrap
From your favorite integrated development environment (IDE) terminal, clone the TFC-RE repository by running this command:

git clone https://github.com/hashicorp/aws-service-catalog-engine-for-tfc.git
cd aws-service-catalog-engine-for-tfc

Configure your IDE with AWS credentials from your Service Catalog hub account. Set the default AWS region to match the region you plan to deploy the TFC-RE.
From your IDE, open the main.tf file in the root of the cloned repository. Update your provider block with the Terraform Cloud API token created in the prerequisites section. To securely use this token, we’re using AWS Secrets Manager data source and making a reference to this value in the provider block.

data "aws_secretsmanager_secret_version" "tfe_token_secret" {
  secret_id = "terraform-cloud-credentials-for-bootstrap"
}

provider "tfe" {
  hostname = var.tfc_hostname
  token    = data.aws_secretsmanager_secret_version.tfe_token_secret.secret_string
}

Continue to edit the main.tf file, and add the Terraform Cloud configuration on the terraform block as shown below. Make sure to change the organization name to your Terraform Cloud organization.

terraform {
  cloud {
    organization = "<<CHANGE WITH YOUR ORG NAME>>"

    workspaces {
      name = "tfc_re_bootstrap"
    }
  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.12.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.5.1"
    }
    tfe = {
      source  = "hashicorp/tfe"
      version = "0.45.0"
    }
  }
}

Next, rename the file terraform.tfvars.example to terraform.tfvars. Open the renamed file and modify the tfc_organization variable according to your Terraform Cloud organization name. The variable token_rotation_internal_in_days has a default value of 30 days; you could modify this parameter as per your security requirements.

tfc_organization = "<<CHANGE WITH YOUR ORG NAME>>"
tfc_team = "aws-service-catalog"
token_rotation_interval_in_days = 30

Save all files and run the command below to login to Terraform Cloud. Follow the prompt on the screen to enter your Terraform Cloud token:

terraform login

After you successfully log in, initialize the workspace by running the command:

terraform init

Now, we are ready to deploy and you can run this command to verify the plan and deploy the resources:

terraform apply

Wait until the deployment is complete; you can monitor the deployment status from the IDE terminal or directly in the Terraform Cloud workspace.

You have successfully completed the Terraform Cloud Reference Engine installation in the AWS Service Catalog hub account. Next, you can try to provision a Terraform-based product.

Provision a Product with Terraform Cloud

TFC-RE deployment includes a sample Service Catalog Terraform Cloud product called service-catalog-example-product; this product launches an Amazon S3 bucket with a randomized bucket name.

Figure 2 – Terraform Cloud product type.

For this test, we’ll launch the product directly from the hub account. In a real-world scenario, you’ll share the portfolio with the spoke accounts and launch it from the spoke account.

To test this new product, follow the steps below:

Navigate to the AWS Service Catalog console.
From the Administration section, select the portfolio called TFC Example Portfolio.
Select the Access tab and choose Grant access.
Grant access to your AWS credentials, and use the AWS Identity and Access Management (IAM) principal in your Service Catalog hub account. You can also experiment by using Principal Name if you plan to share the portfolio to spoke accounts.
Navigate to the Service Catalog Provisioning section and select Products.
Select the service-catalog-example-product-xxxxxx you just granted access earlier.
Select Launch product.
Give the provisioned product a name, or use the Generate name option.
Leave the other parameters as default and add the optional tags as needed.
Select Launch product to continue.
Navigate back to Terraform Cloud and locate the newly-created Terraform workspace.

Figure 3 – New Terraform Cloud project and workspace.

Select the workspace name to monitor the Terraform apply progress.

Figure 4 – Terraform Cloud workspace run status.

Navigate back to Service Catalog provisioned products, refresh the screen, and your new S3 bucket should be fully provisioned.

Figure 5 – Service Catalog provisioned product complete.

You can try to experiment further by updating the provisioned product, for example by changing the Terraform parameters. Service Catalog automatically parses all Terraform variables and presents them under the Parameters section in Service Catalog console. You can inspect the example product variables from this repository.

To complete the test, terminate this test provisioned product:

From Service Catalog provisioned products, select the provisioned product earlier.
Select Action > Terminate to terminate the product.
Wait until the product is terminated successfully.
Navigate to Terraform Cloud and confirm the workspace is deleted from Terraform Cloud as well.

Congratulations, you have successfully provisioned a Service Catalog product using Terraform Cloud.

Cleanup

To avoid incurring future changes, we recommend you remove all resources created within the TFC-RE deployment if you’re no longer using it.

Navigate to the AWS Service Catalog console.
From the Administration section, select the portfolio called TFC Example Portfolio.
Select the Access tab and remove the IAM principals.
From your IDE terminal, run the terraform destroy command and follow the prompt
Wait until all the resources are deleted. You can track this action from the Terraform Cloud in the tfc_re_bootstrap workspace.

Additional Information

You can create a new Terraform Cloud product type in AWS Service Catalog and add it to an existing or new portfolio. You can find the getting started tutorial in the Service Catalo documentation.

There are a couple of considerations to know as you create a new Terraform Cloud product type:

Your Terraform configuration file must be uploaded as a tar.gz file; this is also applicable for Git-based products.
The configuration version supports file sizes up to 950KB.
You can use sub-modules from Terraform Cloud private registry in your configuration.

TFC-RE uses Terraform Cloud Dynamic Provider Credentials (OIDC) to authenticate to the target AWS account. As part of the TFC-RE setup, it automatically creates the OIDC IAM Identity Provider (IdP) in the Service Catalog hub account. For the hub-and-spoke model, you need to create the OIDC IdP in the spoke accounts and the IAM role used for role constraint in the Service Catalog portfolio.

You can find an example configuration to bootstrap the OIDC IdP from this repository.

For each AWS Service Catalog product, TFC-RE automatically creates the Terraform project associated with the product ID. As the Terraform Cloud admin, you can attach team permission to manage the workspace or Sentinel policy at the project level.

To implement that, we recommend you create the Terraform project at the same time you create the Service Catalog product. You can find a sample implementation below:

resource "aws_servicecatalog_product" "example" {
  name = "example-product"
  type = "TERRAFORM_CLOUD"
  # ... other Service Catalog product parameters ...
}

resource "tfe_project" "example" {
  name = aws_servicecatalog_product.example.id
}

# Add team permissions
resource "tfe_team_project_access" "example" {
  access = "read"
  team_id = tfe_team.example.id
  project_id = tfe_project.example.id
}

# Attach policy set
resource "tfe_project_policy_set" "example" {
  policy_set_id = data.tfe_policy_set.aws.id
  project_id = tfe_project.example.id
}

Conclusion

In this post, we showed how to use Terraform Cloud in AWS Service Catalog to provision products. Using this integration, you can enable end users to discover, provision, and operate Terraform-based cloud infrastructure self-service using Service Catalog.

You can also provide end users with a pre-validated catalog of infrastructure and enforce governance through Terraform Cloud features such as team permissions, run tasks, policy sets and more.

To learn more about Terraform Cloud, check out this HashiCorp learn tutorial. Be sure to review the HashiCorp announcement blog and AWS What’s New post to stay up to date with the latest HashiCorp and AWS announcements. You can also find Terraform Cloud in AWS Marketplace.

.

.

HashiCorp – AWS Partner Spotlight

HashiCorp is an AWS Partner and cloud infrastructure automation company that provides the open-source tools Terraform, Vagrant, Packer, Vault, Consul, and Nomad.

Contact HashiCorp | Partner Overview | AWS Marketplace | Case Studies

AWS Partner Network (APN) Blog