AWS Cloud Operations Blog

Use tags to create and maintain Amazon CloudWatch alarms for Amazon EC2 instances (Part 2)

This blog post is the second in a two-part series. Part one of this blog post showed how to deploy and configure the CloudWatchAutoAlarms Lambda function to create a default alarm set and custom alarms for your Amazon Elastic Compute Cloud (Amazon EC2) instances using EC2 instance tags. In this post, I show how you can use AWS Config rules to enforce and remediate the required activation tag for your EC2 instances. In an account with many different workloads and workload owners, it can be difficult to consistently monitor and add alarms to your EC2 instances. If users provision instances without the required activation tag mentioned in the first blog post of this series, then your AWS environment might include EC2 instances without alarms. You can use an AWS Config rule to identify instances that are missing the required activation tag and an AWS Config rule remediation action to add the activation tag to these instances for standard alarm set creation.

Prerequisites

Identify EC2 instances missing the required activation tag for CloudWatchAutoAlarms

AWS Config includes a large collection of predefined rules that you can enable to track configuration compliance and perform remediation for AWS resources. In this walkthrough, I show how to use the required-tags AWS managed rule to track the Create_Auto_Alarms tag used by the CloudWatchAutoAlarms Lambda function.

  1. Open the AWS Config console. If this is the first time you have used AWS Config then choose Setup up AWS Config, otherwise choose Settings. Turn on recording of resource changes for the EC2: Instance resource type. If you are already recording All resources or the AWS EC2 Instance resource type, then you can skip this step.

    The Settings page shows that recording is on. Under resource types to record, the Specific types field displays AWS EC2 Instance.

    Figure 1: Turn on AWS Config recording for EC2 instance resources

  2. From the left navigation pane, choose Rules, and then choose Add rule.
  3. Choose Add AWS managed rule and enter required-tags in the Find rules search box. Select the required-tags rule and choose Next.

    The Add rule page displays an Add AWS managed rule button and a search box. required-tags is entered in the search box and the required-tags predefined rule is displayed and selected below.

    Figure 2: Add an AWS Config rule

  4. Enter required-tags-cloudwatch-auto-alarms for the Name. In Description, enter Identifies EC2 instances missing the Create_Auto_Alarms tag and adds the tag to the EC2 instance.
  5. For Trigger type, use the When configuration changes default option. This performs a check when a new EC2 instance is launched and whenever a setting on any existing EC2 instance changes, such as adding and removing tags.
  6. For Scope of changes, use the default option (Resources).
  7. For Resources, remove all resource types except for AWS EC2 Instance.
  8. For rule parameters, for tag1Key, enter Create_Auto_Alarms. Choose the Remove button next to all other rule parameters so that only the tag1Key parameter remains and then choose Next.

    The Parameters page displays a read-only parameter key named tag1Key and a parameter value with Create_Auto_Alarms. A Remove button is displayed next to the parameter key and value and Next button is displayed in the lower right hand corner.

    Figure 3: Customize the AWS Config rule parameters for the AWS managed required-tags rule

  9. Choose Add rule on the review page to create the customized required-tags-cloudwatch-auto-alarms AWS config rule.

The required-tags-cloudwatch-auto-alarms AWS config rule is now active and AWS config will display any non-compliant Amazon EC2 instances that do not have the Create_Auto_Alarms tag in the AWS Config Rules menu.

Setup AWS Config rule remediation to tag Amazon EC2 instances missing the required activation tag for CloudWatchAutoAlarms

You can create a remediation action to automatically tag Amazon EC2 instances that are missing the Create_Auto_Alarms tag.

  1. From the left navigation pane, choose Rules and then select the required-tags-cloudwatch-auto-alarms rule.
  2. Choose Actions, and then Manage Remediation.
  3. For Select remediation method, choose Automatic remediation. This runs the remediation action you selected without any manual intervention.
  4. In the Retries field, enter 1 retry in 60 seconds. The rule checks compliance for every 60 seconds and retries the remediation if the resource is still noncompliant.
  5. For Choose remediation action, select AWS-PublishSNSNotification. AWS Config supports any AWS Systems Manager Automation runbook as a remediation action. The AWS-PublishSNSNotification runbook enables you to integrate your remediation action with many different endpoints using Amazon Simple Notification Service (SNS), like AWS Lambda functions.

    On the Choose remediation action page, AWS-PublishSNSNotification is selected for the remediation action. The auto remediation option is set to Yes. Retries is set to 1 retries in 60 seconds.

    Figure 6: Configure the AWS Config rule with AWS-PublishSNSNotification

  6. Skip the Rate Limits section to accept the default values.
  7. For Resource ID parameter, choose Message. The EC2 instance ID is used for the Message parameter in the AWS-PublishSNSNotification Systems Manager Automation runbook.
  8. Deploy this sample Amazon SNS CloudFormation template to create an Amazon SNS topic for this AWS Config rule. When the stack is deployed, enter the created SNS topic Amazon Resource Name (ARN) in the TopicArn parameter value. You can retrieve the ARN in the CloudFormation Outputs tab for the CloudFormation stack. This SNS topic will be notified when an EC2 instance is found that is missing the Create_Auto_Alarms tag.
  9. Deploy this sample Systems Manager automation role CloudFormation template to create an AWS Systems Manager automation role with Amazon SNS publish permissions. Enter the created SNS topic ARN from step 8 for the SNSPublishMessageTopicARN CloudFormation parameter. When the stack is deployed, enter the created AWS Identity and Access Management (IAM) role ARN in the AutomationAssumeRole* parameter value. You can retrieve the ARN in the CloudFormation Outputs tab for the CloudFormation stack.
  10. Choose Save Changes to activate the remediation.

    In Parameters, the AutomationAssumeRole and TopicArn key and value sets for the AWS-PublishSNSNotification remediation action are displayed

    Figure 7: Integrate the AWS Config rule with your custom Amazon SNS notification

Currently, there are no subscriptions to the SNS topic specified for the rule, so messages sent to the topic do not have any effect. You can deploy this sample AWS Lambda CloudFormation template to deploy a simple AWS Lambda function that adds tags to an EC2 instance using tags that you specify. The CloudFormation template deploys a Lambda function named EC2Tagger, subscribes it to an SNS topic you specify in the SNSTopicArn CloudFormation parameter, and gives the topic permissions to invoke the Lambda function. You can use the ARN for the SNS topic you created in step 8 for the SNSTopicArn CloudFormation parameter.

When the AWS Config rule is triggered because of a missing tag on an EC2 instance, a new message is sent to the Amazon SNS topic you specified with the instance ID in the Message field. The EC2Tagger Lambda function is called by the SNS topic and the EC2 instance is tagged with the tag key, Create_Auto_Alarms. You can then stop and start the instances at a convenient time (for example, a maintenance window) to create standard Amazon CloudWatch alarms as described in the first blog post of this series.

You can test the AWS Config rule remediation by launching an Amazon EC2 instance into your account without the Create_Auto_Alarms tag. Follow the directions in Part 1 of the blog post for an example of how to launch an EC2 instance with the CloudWatch agent from the AWS EC2 console. Once your AWS Config Rule remediation is executed, the missing Create_Auto_Alarms tag will be added to the instance. You will still need to stop and start the instance in order for the CloudWatchAutoAlarms Lambda function to execute and create the standard alarm set for the EC2 instance.

Cleanup

To avoid incurring additional charges in your account, clean up the resources you created.

Conclusion

In large AWS environments where many users use different methods for provisioning instances, it becomes difficult to consistently monitor and add CloudWatch alarms to your instances. In this blog post, I showed how to use AWS Config rules to identify EC2 instances that are missing the required tags for standard alarm set creation with the open source CloudWatchAutoAlarms Lambda function. AWS Config rules allow you to remediate violated rules. The EC2Tagger AWS Lambda function included in this post can be used to retag an instance so that standard alarms are created. You can modify the EC2Tagger function to perform other actions or create multiple remediations using any of the supported Amazon SNS endpoints.

About the Author

Khurram Nizami

Khurram Nizami is an Operations Management Consultant at Amazon Web Services (AWS). Khurram is passionate about helping people build innovative solutions using technology. In his free time, Khurram enjoys hiking, nature, DIY projects, and travel.