Automating life-cycle management for ephemeral resources using AWS Service Catalog

Enterprises deploy AWS resources and services daily to support different business objectives.

For example:

A data scientist might like to create an EMR cluster for a job that should not take longer than one week.
A sales engineer needs a demo environment for two days.
A marketing application owner wants a marketing application to run for nine weeks.
A QA engineer would like to run a QA task for five days.

All of these tasks have time bounds and definitions that end users would like to manage and remove these resources when they are no longer needed. To do this, they need a process that automatically terminates unneeded resources.

In this post, you will learn how to enable end users to pick expiration times for their deployed resources on AWS. Additionally, you learn how to set a hard time limit of 30 days on these ephemeral resources.

This solution uses the following AWS services. Most of the resources are set up for you with an AWS CloudFormation stack:

Background

Here are some of AWS Service Catalog concepts referenced in this post. For more information, see Overview of AWS Service Catalog.

A product is a blueprint for building the AWS resources to make available for deployment on AWS, along with the configuration information. Create a product by importing an AWS CloudFormation template, or, in case of AWS Marketplace-based products, by copying the product to AWS Service Catalog. A product can belong to multiple portfolios.
A portfolio is a collection of products, together with the configuration information. Use portfolios to manage user access to specific products. You can grant portfolio access for an AWS Identity and Access Management (IAM) user, IAM group, or IAM role level.
A provisioned product is an AWS CloudFormation stack; that is, the AWS resources that are created. When an end-user launches a product, AWS Service Catalog provisions the product from an AWS CloudFormation stack.
Constraints control the way that users can deploy a product. With launch constraints, you can specify a role that AWS

Solution overview

The following diagram maps out the solution architecture.

Solution flow description

This solution uses AWS Service Catalog for product creation, Lambda and CloudWatch for the termination process, and DynamoDB for state management. It also uses SES for notifications to the end user.

1. Admin experience

The AWS administrator uses the AWS CloudFormation console to launch the setup template provided as part of this post.

As part of the deployment process, AWS CloudFormation creates the following resources:

A CloudWatch rule
A Lambda function
A sample AWS Service Catalog portfolio with a sample AWS Service Catalog product
A DynamoDB table

2. End-user experience

The end user could be a data scientist or QA engineer. They log in to the AWS Service Catalog console, pick a product such an EMR cluster or a product from the Service Catalog reference architecture, choose how long the product should be active, and launch the product.

Behind the scenes, invisible to the end user, CloudWatch detects that the product being deployed is configured for automatic termination and triggers a Lambda function. The Lambda function reads the product information and stores the info in a DynamoDB table.

3. Scheduled process

Invisible to the end user, a CloudWatch rule is triggered at certain intervals selected by the AWS administrator during the setup process. It triggers a Lambda function.

The Lambda function queries the DynamoDB table and gets a list of AWS Service Catalog products that have reached the end of their subscription time. It terminates those products.

The owner of the AWS Service Catalog product is notified through SES that the service has been terminated.

Walkthrough

Configure an environment.
Launch the end user Service Catalog product with auto termination.
Schedule a process to check for the provisioned product end time.
Configure AWS Service Catalog products with auto termination.

Configure an environment

Use an Amazon S3 bucket to upload your configuration files from AWS CloudFormation and Lambda.

To get the setup material:

Download the sc_lscautoterminate.zip file with the configuration content.
Unzip the contents and save them to a folder. Note the folder’s location.

To create your S3 bucket:

Log in to your AWS account as an administrator. Ensure that you have an AdministratorAccess IAM policy attached to your login because you’re going to create AWS resources, including IAM roles and users.
In the S3 console, create a bucket. Leave the default values except as noted.
- For Bucket name, enter scautoterminate-<accountNumber>.

To upload content to the new bucket:

In the S3 console, select your new bucket, and choose Upload, Add files.
Navigate to the folder that contains the configuration content. Select all the files and choose Open. Leave the default values except as noted.
After the Review page, from the list of files, select the sc_admin_setup_autoterminate.json file.
Right-click the link under Object URL and choose Copy link address.

To launch the configuration stack:

In the AWS CloudFormation console, choose Create Stack, Amazon S3 URL, paste the URL that you just copied, and then choose Next.
On the Specify stack details page, specify the following:
- Stack name: scautoterminateSetup-<accountNumber>
- S3Bucket: scautoterminate-<accountNumber>
- SCEndUser: The current user name.
- CheckFrequency: The interval of when to check for products end of life (10,20,30) minutes
On the Review page, check the box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names, and choose Create.
After the status of the stack changes to CREATE COMPLETE, select the stack and choose Outputs to see the output.
Find the ServiceCatalog entry and copy the URL value.

Congratulations! You have completed the setup. Now, test it by following the launch the end-user Service Catalog product with auto termination process.

Launch the end-user Service Catalog product with auto termination

To launch an AWS Service Catalog product with auto termination:

Log in to the AWS Management Console as the SCEndUser.
In the AWS Service Catalog console, in the left navigation pane, choose Products list.
On the Products list page, choose the AWS Service Catalog sample product BucketAutoTerminate that was configured during the setup process.
Choose Launch product.
On the Product Version page, for Name, enter scbucket10min, and choose Next.
On the Parameters page, set the following values:
- AutoTerminate – True.
- DurationDays – How long the product should run in days.
- DurationHours – How long the product should run in hours.
- DurationMinutes – How long the product should run in minutes.
- Action – Terminate or Notify.
- ContactEmail – Your email address.
- BucketName – The bucket name (use the default, as the system always creates a unique name).
On the TagOptions page, choose Next.
On the Review page, choose Launch.

Congratulations, you have deployed an AWS Service Catalog product with an auto termination configuration.

The product has now been successfully deployed. After the product is launched, the end user is sent an email with the end time information of the provisioned product, as in the following image.

Also, invisible to the end user, a CloudWatch rule is triggered that detects when the product being launched is configured for auto termination and stores the product information in a DynamoDB table. This information is used later when the scheduled process is executed.

Schedule a process to check for the provisioned product end time

An automated process is triggered at the interval chosen at the setup time. The process checks the current time against the end time of the provisioned products. If the process finds a provisioned product that has reached its end time, the process performs one of the following actions:

If the product is configured with an end action of Notify, the end user gets an email notification and the product remains active.
If the product is configured with an end action of Terminate, the product is terminated and the end user gets an email notification, similar to the following image.

Configure AWS Service Catalog products with auto termination

Here’s how you configure products with auto termination:

Create or modify the existing CloudFormation template.
Create a new AWS Service Catalog product version

Create or modify the existing CloudFormation template

With a text editor, edit the CloudFormation template on which the AWS Service Catalog product is based.

Add the following parameters and Metadata section to the file, and save the file.

"ContactEmail": {
      "Default": "email@emai.com",
      "Description": "Email of owner",
      "Type": "String"
    },
    "DurationDays": {
      "Default": "0",
      "AllowedValues": [
        "0",
        "2",
        "7",
        "30",
        "60",
        "90"
      ],
      "Description": "Duration Days",
      "Type": "String"
    },
    "DurationHours": {
      "Default": "0",
      "AllowedValues": [
        "0",
        "1",
        "2",
        "3",
        "4",
        "5"
      ],
      "Description": "Duration Hours",
      "Type": "String"
    },
    "DurationMinutes": {
      "Default": "10",
      "AllowedValues": [
        "10",
        "20",
        "30",
        "40",
        "50",
        "60"
      ],
      "Description": "Duration Minutes",
      "Type": "String"
    },
    "AutoTerminate": {
      "Default": "True",
      "AllowedValues": [
        "True",
        "False"
      ],
      "Description": "Should the product automatically terminate?",
      "Type": "String"
    },
    "Action": {
      "Default": "Terminate",
      "AllowedValues": [
        "Terminate",
        "Notify",
        "Other"
      ],
      "Description": "What action should be taken at the end of the service?",
      "Type": "String"
    },
 
  },
  "Metadata": {
    "AWS::CloudFormation::Interface": {
      "ParameterGroups": [
        {
          "Label": {
            "default": "End Information"
          },
          "Parameters": [
            "AutoTerminate",
            "DurationDays",
            "DurationHours",
            "DurationMinutes",
            "Action",
            "ContactEmail"
          ]
        }
      ]
    }
  }

Create a new AWS Service Catalog product version

Log in to the AWS Service Catalog console as an AWS Service Catalog admin user.
In the left navigation pane, choose Product list and select the product to which to add a new version.
Choose CREATE NEW VERSION.
Choose Browse and select the file that you saved earlier.
For Version title, enter “V9 auto terminate”.
For Description, enter a description.
Choose SAVE.

You can now deploy the auto-terminate version by following the instructions in Step 2.

Cleanup process.

To avoid incurring cost, please delete resources that are not needed. You can terminate the Service Catalog product deployed the by selecting Action then, Terminate.

Conclusion

In this post, you learned an easy way to terminate resources automatically on AWS with AWS Service Catalog. You also saw how there’s an extra layer of governance and control when you use AWS Service Catalog to deploy resources to support business objectives.

About the Author

Kenneth Walsh is a New York-based Solutions Architect focusing on AWS Marketplace. Kenneth is passionate about cloud computing and loves being a trusted advisor for his customers. When he’s not working with customers on their journey to the cloud, he enjoys cooking, audio books, movies, and spending time with his family and dog.

AWS Cloud Operations Blog