Proactively keep resources secure and compliant with AWS CloudFormation Hooks

Organizations want their developers to provision resources that they need to build applications while maintaining compliance with security, operational, and cost optimization best practices. Most solutions today inform customers about noncompliant resources only after those resources have been provisioned. These noncompliant resources exist until they are deleted or modified and increase security risk, operational overhead, and cost for customers. Customers that want a more proactive compliance enforcement system would have to build their own control mechanism, and make sure that resource provisioning is subjected to the compliance checks. However, many organizations don’t have the internal expertise or development capacity to build these comprehensive systems.

This led us to create AWS CloudFormation Hooks, a new AWS CloudFormation feature that lets customers run code before creating, updating, or deleting a resource. The result of the code can be either to trigger a warning message, or prevent the deployment of a resource. Customers choose CloudFormation Hooks to provide the automatic and proactive enforcement of business requirements through a series of checks during provisioning. Resources are only provisioned if they pass these checks, effectively lowering security and compliance risks, reducing overhead of fixing compliance issues, and optimizing costs.

CloudFormation Hooks is a supported extension type in the AWS CloudFormation registry. The registry makes it easier to distribute and consume Hooks either publicly or privately, and supports versioning. The registry also supports resources and modules extension types.

In this post, we will first show you how to activate and configure hooks from the public registry. Second, we will create and deploy a hook to your private registry that only lets an Amazon Elastic Compute Cloud (EC2) instance be deployed if it is using the compliant Amazon Machine Image (AMI). The ImageID of the required AMI will be stored in a parameter in the Parameter Store, a capability of AWS Systems Manager (SSM).

Key terms and concepts

Hook – A hook contains code that is invoked immediately before CloudFormation creates, updates, or deletes specific resources. Hooks can inspect the resources that CloudFormation is about to provision. If Hooks find any resources that don’t comply with your organizational guidelines defined in your hook logic, then you may choose to either WARN users, effectively in observe mode, or FAIL, effectively preventing CloudFormation from continuing the provisioning process.

Hook Targets – Hook targets are the CloudFormation resources that you want to run a hook against. Targets can be general resources that CloudFormation supports, or third-party resources in the registry. As a hook author, you specify target(s) while authoring a hook. For example, you can author a hook targeting the AWS::S3::Bucket resource. A hook supports multiple targets, and there is no limit on the number of resource targets that a hook supports.

Target Invocation Point – Invocation points are the exact point in provisioning logic where hooks run. CloudFormation currently supports PRE (before) invocation point. This means that you can write a hook that runs before the provisioning logic for the target is started. For example, a hook with PRE invocation point for an Amazon Simple Storage Service (Amazon S3) target runs before the service starts provisioning an Amazon S3 bucket in your account.

Target Action – Target Action is the type of operation that triggers a hook. For example, the Amazon S3 CloudFormation resource target supports CREATE, UPDATE, and DELETE actions. When a hook for CREATE action on an Amazon S3 target is created, it only runs at the creation of an Amazon S3 bucket.

Hook Handlers – The combination of Invocation Point and Action make an exact point where a hook runs. As a hook author, you write handlers that host logic for these specific points. For example, a PRE invocation point with CREATE action makes a preCreate handler. Code within the handler runs at any time that a matching target and service are performing an associated action.

Activating a Hook from the Public Registry

To get started quickly, you can activate a sample hook from the Public Registry.

Go to the AWS CloudFormation console and click Public extensions from the navigation bar. This will take you to the Registry: Public extensions home page. Under Filter, click Extension type Hooks and Publisher Third party to view third party Hook extensions. These sample hooks are also available in the AWS CloudFormation Hooks sample repository on Github. Select AWSSamples::Ec2ImageIdCheckSsm::Hook from the list to view details.

You can view the Hook documentation by selecting View Documentation under the Hook name. The Readme provides details on the schema, Configuration JSON, description, and more. Copy the sample Configuration JSON from the Readme into a notepad to use in a later step. Click Activate to publish the hook to your account.

Enter the Execution role ARN of the IAM execution role for CloudFormation to assume when invoking the hook in your account and region. In order for CloudFormation to assume the execution role, the role must have a trust policy defined with CloudFormation. Create a new role in the IAM Console with the following custom trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "resources.cloudformation.amazonaws.com",
                    "hooks.cloudformation.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account"
                },
                "StringLike": {
                    "aws:SourceArn": [
                        "arn:aws:cloudformation:region:account:type/hook/AWSSamples-Ec2ImageIdCheckSsm-Hook/*"
                    ]
                }
            }
        }
    ]
}

AWSSamples::Ec2ImageIdCheckSsm::Hook requires an execution role with the following permissions:

ssm:GetParameter

Under Logging config, specify logging configuration information for the hook, if desired. For example:

{
    "logRoleArn": "arn:aws:iam::account:role/rolename",
    "logGroupName": "log-group-name"
}

The logging role requires the custom trust policy in Step 3 and the following permissions:

cloudwatch:ListMetrics
cloudwatch:PutMetricData
logs:CreateLogStream
logs:DescribeLogGroups
logs:DescribeLogStreams
logs:CreateLogGroup
logs:PutLogEvents

Review the remainder of the settings and click Activate extension to proceed.

Once a public hook is activated in your account, you must set the configuration before it will apply against your stack operations. Configuring a hook allows you to turn the hook on or off (TargetStacks = ALL or NONE), set FailureMode as FAIL or WARN, and set any type configuration properties specific to the hook. For more information on TargetStacks, and FailureMode options, refer to Hook configuration schema in the AWS CloudFormation Hooks User Guide.

SsmKey is a type configuration property specific to this hook. It is used to retrieve the desired AMI ImageID during the hook’s run time. Set this property to the Parameter Store parameter that will contain the value of the compliant AMI’s ImageID. Modify the sample Configuration JSON you copied in Step 2 to match the following:

{
    "CloudFormationConfiguration": {
        "HookConfiguration": {
            "TargetStacks":"ALL",
            "FailureMode":"FAIL",
            "Properties": {
                "SsmKey": "compliant-imageid-x86"
            }
        }
    }
}

Set the Configuration alias to default, and paste the Configuration JSON you modified in the previous step to Configuration JSON. Click Configure extension to proceed.

Before you launch a stack to trigger the hook, you need to create the Parameter Store parameter that contains the value of the compliant AMI’s ImageID. You can do this using the AWS Systems Manager Parameter Store Console or AWS Command Line Interface (CLI). The parameter name should match the SsmKey set in Step 6. The ImageID ami-0e5b6b6a9f3db6db8 represents the Amazon Linux 2 (HVM) (64-bit x86) AMI in us-west-2 at the time that this post was written.

$ aws ssm put-parameter --name "compliant-imageid-x86" --value "ami-0e5b6b6a9f3db6db8" --type String

Refer to Testing the hook later in this post for steps to test the hook.
To disable the hook in your account, modify the hook’s configuration property TargetStacks to NONE in Set configuration.

Authoring and publishing your own Hook

Prerequisites

This post assumes that you’re familiar with AWS CloudFormation templates, Python, and Docker. For this walkthrough, you must have an AWS account.

1. Set up your development environment by completing the following steps:
  1. Install Python 3.6 or later by either downloading Python or using your operating system’s package manager.
  2. Use the following command to install both the cloudformation-cli (cfn) and Python language plugin. If you already have these packages installed, then you must upgrade them to get the new Hooks functionality.

Install:

$ pip3 install cloudformation-cli cloudformation-cli-python-plugin

Upgrade:

$ pip3 install --upgrade cloudformation-cli cloudformation-cli-python-plugin

Make sure AWS Command Line Interface (CLI) is installed and configured with your AWS credentials. This post uses the us-west-2 Region.
```
$ aws configure
Default region name [us-east-1]: us-west-2
```
Create the Parameter Store parameter that will contain the value of the compliant AMI’s ImageID. The ImageID ami-0e5b6b6a9f3db6db8 represents the Amazon Linux 2 (HVM) (64-bit x86) AMI in us-west-2 at the time that this post was written.
```
$ aws ssm put-parameter --name "compliant-imageid-x86" --value "ami-0e5b6b6a9f3db6db8" --type String
```

Permissions Required

In addition to CloudFormation create/delete/update stack permissions, you need the following CloudFormation permissions:

Initiate a project

Use the cfn init command to create your hook project and generate the files that it requires. Make sure you run the following steps inside of the folder that you created:
```
$ mkdir myfirsthook
$ cd myfirsthook
$ cfn init
```
The cloudformation-cli prompts you with the option to create a resource, module, or hook. Enter h for hook.
```
Initializing new project
Do you want to develop a new resource(r) or a module(m) or a hook(h)?.
>> h
```
The cloudformation-cli prompts you for the name of the hook type, which maps to the Type attribute for resources in a CloudFormation template. For this example, use Demo::Testing::MyFirstHook
```
What's the name of your hook type?
(Organization::Service::Hook)
>> Demo::Testing::MyFirstHook
```
Select the appropriate language plugin. Java, Python 3.6, and Python 3.7 are supported. For this walkthrough, select Python37:
```
Select a language for code generation:
[1] java
[2] python36
[3] python37
(enter an integer): 
>> 3
```
Choose whether to use Docker for platform-independent packaging of Python dependencies. Although not required, this is highly recommended to make development easier.
```
Use docker for platform-independent packaging (Y/n)?
This is highly recommended unless you are experienced 
with cross-platform Python packaging.
>> Y
```

Initiating the project generates files needed to develop a hook. Boiler plate code for your handlers and models is included in the src folder.

$ ls -1
README.md
demo-testing-myfirsthook.json
hook-role.yaml
requirements.txt
rpdk.log
src
template.yml

Creating your first CloudFormation hook

A hook includes a hook specification represented by a JSON schema, and handlers that will be invoked at each hook invocation point. Once you create these, the hook must be registered and enabled in your account. The following steps walk you through this process.

Step 1 Modeling a hook

The first step in creating a hook is modeling the hook, which involves crafting a schema that defines the hook, its properties, and their attributes. When you create your hook project using the cfn init command, one of the files created is an example hook schema as a JSON-formatted text file <hook-name>.json. Use this schema file as a starting point for defining the shape and semantics of your hook. For more details on the schema, please refer to Hook schema property reference in the AWS CloudFormation Hooks User Guide.

Use the hook schema to specify the handlers that you want to implement. In this scenario, you are going to implement the preCreate handler for the AWS::EC2::Instance target. This means that any time an EC2 instance is created using CloudFormation, the hook will be invoked. You do not need to create a preUpdate handler for this compliance check, because when an EC2 instance’s AMI ImageID is changed, CloudFormation will not run an update resource. Instead, CloudFormation deletes the resource and create a new one with the updated AMI ImageId.

The hook needs AWS Identity and Access Management (IAM) permissions to get the compliant ImageID parameter from the Parameter Store, add the ssm:GetParameter permission to the preCreate handler.

Open demo-testing-myfirsthook.json and replace the content with the following:

{
    "typeName": "Demo::Testing::MyFirstHook",
    "description": "Validates that EC2 Instances are using the correct ImageID from SSM",
    "sourceUrl": "https://example.com/my-repo.git",
    "documentationUrl": "https://example.com/documentation",
    "typeConfiguration": {
        "properties": {
            "SsmKey": {
                "description": "The key to get the ImageID from",
                "type": "string"
            }
        },
        "additionalProperties": false
    },
    "required": [
        "SsmKey"
    ],
    "handlers": {
        "preCreate": {
            "targetNames": [
                "AWS::EC2::Instance"
            ],
            "permissions": ["ssm:GetParameter"]
        }
    },
    "additionalProperties": false
}

Step 2 Generating the hook project package

The next step is generating your hook project package. The cloudformation-cli will create empty handler functions, each of which corresponds to a specific hook invocation point in the target lifecycle as defined in the hook specification.

$ cfn generate
Generated files for Demo::Testing::MyFirstHook

Step 3 Write the hook handler code

It is now time to write your hook code. In this walkthrough you will implement the preCreate invocation point handler. The code will retrieve the compliant AMI ImageID from Parameter Store, and check whether it matches the ImageID property for the AWS::EC2::Instance resource. If the ImageID does not match, then Hooks will return a failure notice.

Open the handlers.py file, located in the src/demo_testing_myfirsthook folder.
Replace the entire contents of the handlers.py file with the following code:

import logging
from typing import Any, MutableMapping, Optional

from cloudformation_cli_python_lib import (
    BaseHookHandlerRequest,
    HandlerErrorCode,
    Hook,
    HookInvocationPoint,
    OperationStatus,
    ProgressEvent,
    SessionProxy,
    exceptions,
)
from .models import HookHandlerRequest, TypeConfigurationModel

# Use this logger to forward log messages to CloudWatch Logs.
LOG = logging.getLogger(__name__)
TYPE_NAME = "Demo::Testing::MyFirstHook"

# Set the logging level
LOG.setLevel(logging.INFO) #or LOG.setLevel(logging.DEBUG)

hook = Hook(TYPE_NAME, TypeConfigurationModel)
test_entrypoint = hook.test_entrypoint

def _validate_ec2_instance_imageid(progress, target_name, resource_properties, ssm_key, session):
    try:
        if resource_properties:
            LOG.debug(f"DEBUG Details of resource_properties: {resource_properties}")

            # Get the name of the EC2 instance based on tag
            instance_tags = resource_properties.get("Tags")
            instance_name = None
            if instance_tags is not None:
                instance_name_tag = next(filter(lambda x: x['Key'] == 'Name', instance_tags))
                instance_name = instance_name_tag.get("Value")

            if instance_name is None:
                instance_name = "NameTagNotSet"
                LOG.debug(f"DEBUG Could not find the Tag 'Name'. Setting the EC2 Instance name to {instance_name}")
            else:
                LOG.debug(f"DEBUG EC2 Instance has the following name tag {instance_name}")


            # Get the ImageID of the AMI for the EC2 instance based on tag
            instance_imageid = resource_properties.get("ImageId")
            LOG.debug(f"DEBUG EC2 Instance has the following ImageID {instance_imageid}")

            # Get the expected ImageID from SSM Parameter Store
            client = session.client('ssm')
            expected_imageid_ssm = client.get_parameter(
                Name=ssm_key,
                WithDecryption=True
            )
            expected_imageid = expected_imageid_ssm.get("Parameter", {}).get("Value")

            LOG.debug(f"DEBUG Verifying EC2 Instance ImageId for target {target_name}, expecting target to have ImageId {expected_imageid}.")

            if expected_imageid == instance_imageid:
                progress.status = OperationStatus.SUCCESS
                progress.message = f"Successfully invoked HookHandler for target {target_name} with name: {instance_name}. ImageId {instance_imageid} matches the required AMI"
            else:
                progress.status = OperationStatus.FAILED
                progress.message = f"Failed to verify ImageId for EC2 instance {instance_name}. Expected ImageId to be {expected_imageid}, actual ImageId is {instance_imageid}."
                progress.errorCode = HandlerErrorCode.NonCompliant

        else:
            progress.status = OperationStatus.FAILED
            progress.message = f"Failed to verify ImageId for target {target_name}."
            progress.errorCode = HandlerErrorCode.InternalFailure

    except TypeError as e:
        progress.status = OperationStatus.FAILED
        progress.message = f"was not expecting type {e}."
        progress.errorCode = HandlerErrorCode.InternalFailure

    LOG.info(f"Results Message: {progress.message}")

    return progress


@hook.handler(HookInvocationPoint.CREATE_PRE_PROVISION)
def pre_create_handler(
        session: Optional[SessionProxy],
        request: HookHandlerRequest,
        callback_context: MutableMapping[str, Any],
        type_configuration: TypeConfigurationModel
) -> ProgressEvent:
    target_name = request.hookContext.targetName
    progress: ProgressEvent = ProgressEvent(
        status=OperationStatus.IN_PROGRESS
    )

    # Make sure this hook is running against the expected resource type
    if "AWS::EC2::Instance" == target_name:
        LOG.info(f"Successfully invoked PreCreateHookHandler for target {target_name}")
        LOG.debug(f"DEBUG SSM Parameter Store Key location for compliant ImageID: {type_configuration.SsmKey}")
        return _validate_ec2_instance_imageid(progress, target_name, request.hookContext.targetModel.get("resourceProperties"), type_configuration.SsmKey, session)
    else:
        return ProgressEvent.failed(HandlerErrorCode.InternalFailure, f"Unknown target type: {target_name}")

Step 4 Register the hook

Before you submit your hook, package your project without registering it to the private registry to check for any packaging errors.

Note: Please make sure that Docker is running for this step, otherwise you will receive a “Unhandled exception” error.

$ cfn submit --dry-run

If the dry run completes successfully, you may now submit your hook to the registry. The next command will call register type API to register your hook. It will keep polling for registration status until it is finished.

$ cfn submit --set-default

If the response contains {‘ProgressStatus’: ‘COMPLETE’}, then your hook has been successfully registered.

Updating a hook

You may continue making changes to your hook by updating handler code and repeating steps from Register the hook. This will upload the latest version to the registry.

Step 5 Enable the hook

Complete the following steps to enable the hook in your account.

Verify that you can list your hook:
```
$ aws cloudformation list-types
```
Get the type ARN from this output for your hook and save it:
```
$ export HOOK_TYPE_ARN=your-hook-arn-non-versioned
```
Enable the hook by modifying its type configuration properties. The three configuration properties are TargetStacks, FailureMode, and SsmKey. For more information on TargetStacks, and FailureMode options, refer to Hook configuration schema in the AWS CloudFormation Hooks User Guide. SsmKey is a user-defined property that was created in Step 1 Modeling a hook. Set this property to the compliant-imageid-x86 parameter that was configured in the prerequisites section.
```
$ aws cloudformation set-type-configuration \
 --configuration "{\"CloudFormationConfiguration\":{\"HookConfiguration\":{\"TargetStacks\":\"ALL\",\"FailureMode\":\"FAIL\",\"Properties\":{\"SsmKey\": \"compliant-imageid-x86\"}}}}" \
--type-arn $HOOK_TYPE_ARN 
```

Note: If you activate hooks from the public registry, you must set the type configuration to ensure the hooks apply to all stacks.

Testing the hook

In this section, you will create a CloudFormation template that defines resources for two EC2 instances, one with the compliant ImageID, and one with a non-compliant ImageID. You will then create a stack using this template to verify the hook runs.

Step 1 Run the hook in a stack

Create a new CloudFormation template named newec2instances.yml and insert the following code:

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  EC2InstanceCompliant:
    Type: AWS::EC2::Instance
    Properties: 
      InstanceType: t2.micro
      SecurityGroups: [!Ref 'InstanceSecurityGroup']
      ImageId: ami-0e5b6b6a9f3db6db8
      Tags:
        - Key: Name
          Value: EC2InstanceCompliant
  EC2InstanceNonCompliant:
    Type: AWS::EC2::Instance
    Properties: 
      InstanceType: t2.micro
      SecurityGroups: [!Ref 'InstanceSecurityGroup']
      ImageId: ami-03d5c68bab01f3496
      Tags:
        - Key: Name
          Value: EC2InstanceNonCompliant
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Empty Security Group
      Tags:
        - Key: Name
          Value: SG-EC2-Empty

Create a stack using this template:

$ aws cloudformation create-stack --stack-name my-first-hook-stack --template-body file://newec2instances.yml

View progress using the describe-stack-events AWS CLI or in the AWS Cloudformation console.
```
$ aws cloudformation describe-stack-events --stack-name my-first-hook-stack
```
It is expected that the stack will fail to create. Go through the stack events to find the event that caused the failure. You should see an event from CloudFormation hooks that provides information as to why the hook failed.

        {
            "StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/my-first-hook-stack/65f61a60-3e61-11ec-8518-0651e37eda47",
            "EventId": "EC2InstanceNonCompliant-7c4247fc-5c7e-4a12-9436-98efec752de2",
            "StackName": "my-first-hook-stack",
            "LogicalResourceId": "EC2InstanceNonCompliant",
            "PhysicalResourceId": "",
            "ResourceType": "AWS::EC2::Instance",
            "Timestamp": "2021-11-05T17:54:33.988000+00:00",
            "ResourceStatus": "CREATE_IN_PROGRESS",
            "HookType": "Demo::Testing::MyFirstHook",
            "HookStatus": "HOOK_COMPLETE_FAILED",
            "HookStatusReason": "Hook failed with message: Failed to verify ImageId for EC2 instance EC2InstanceNonCompliant. Expected ImageId to be ami-0e5b6b6a9f3db6db8, actual ImageId is ami-03d5c68bab01f3496.",
            "HookInvocationPoint": "PRE_PROVISION",
            "HookFailureMode": "FAIL"

Cleanup

To clean up the resources created in your account, perform the following steps:

Disable the hook in your account by changing the hook’s configuration property TargetStacks to NONE.

$ aws cloudformation set-type-configuration \
 --configuration "{\"CloudFormationConfiguration\":{\"HookConfiguration\":{\"TargetStacks\":\"NONE\",\"FailureMode\":\"FAIL\",\"Properties\":{\"SsmKey\": \"compliant-imageid-x86\"}}}}" \
--type-arn $HOOK_TYPE_ARN

Deregister the hook from the registry.

$ aws cloudformation deregister-type --type HOOK --type-name Demo::Testing::MyFirstHook

Delete the CloudFormation stack demo-testing-myfirsthook-role-stack. This stack was created when you registered your hook. This stack provisions an IAM role to be assumed by CloudFormation during Hook operations.
(Optional) Delete the CloudFormation stack CloudFormationManagedUploadInfrastructure. This stack was created as part of the first use of cfn submit. This stack provisions all of the infrastructure required to upload artifacts to CloudFormation as you create future resource types. It includes Amazon S3 buckets, an IAM role, and an AWS Key Management Service (AWS KMS) key that are reused as you build other resource types. We did this for your convenience, to avoid having to create and maintain multiple buckets, roles, and keys as you create multiple resource types. If you need to delete the stack, then make sure that the buckets are empty.

Tips and recommendations

This post only covers CloudFormation Hooks features that are related to the creation of the sample hook in the preceding walkthrough. There are other features and capabilities covered in the AWS CloudFormation Hooks User Guide. Consider the following tips and recommendations as you learn more about Hooks:

Hooks can be published to multiple accounts using StackSets. Review Publishing your extension in multiple Regions using StackSets for more information.
When a warning or failure message is sent back to CloudFormation stack events, it must be explicit so that the developer understands the reason why the resource did not pass.
Start off with the hook in observe mode by setting FailureMode to WARN to monitor the impact before enforcing the rules and preventing resource creation.
Enable logging in to CloudWatch Logs by setting LOG setLevel to either logging.INFO or logging.DEBUG in the hook handler code.
It is possible to subscribe and get your hook events in EventBridge by creating an event-bridge rule to subscribe to hook invocation events.
CloudFormation Hooks runs against all CloudFormation stacks that may be created by CDK, SAM, AWS Amplify, AWS Elastic Beanstalk, etc.

What about templates authored in CDK or SAM?

AWS Cloud Development Kit (CDK) and AWS Serverless Application Model (SAM) authored templates synthesizes CloudFormation templates and resources before deployment. CloudFormation Hooks is a CloudFormation feature, which means hooks will trigger on the applicable resources defined in the hook schema. If you are using CDK or SAM, then you do not need to take any actions in your templates to trigger a hook.

Conclusion

This post covers a CloudFormation compliance scenario where you can proactively evaluate resource configurations with CloudFormation Hooks during stack creation. You can either display a warning message or prevent resources from being provisioned if they do not pass these checks. This effectively lowers security and compliance risks, reduces the overhead of fixing compliance issues, and optimizes costs. We expect the community to contribute hooks via GitHub, which further increases Hook’s value proposition. We look forward to learning how customers use Hooks in new scenarios, and your continued feedback so that we can improve it.

AWS Cloud Operations Blog