An effective patch management is foremost for maintaining system security, reliability, and compliance across your IT infrastructure. AWS Systems Manager (SSM) provides a comprehensive patching solution, enabling you to automate the deployment of operating system updates to your nodes deployed on AWS, on-premises, and multicloud environments. However, as your organization scales, tracking and reporting on patching operations can become increasingly challenging.

Traditionally, monitoring patching operations has been a manual and time-consuming task. To address this challenge, AWS Systems Manager offers features, including Inventory, Explorer and Compliance to report the overall compliance of a node. Customers often request a more centralized approach to storing historical patching data across multiple accounts and receiving detailed email notifications.

In this blog post, we’ll explore how to automate the creation and delivery of patching reports, streamlining the process of tracking patching operations. By leveraging AWS services such as AWS Lambda, Amazon EventBridge, AWS Step Functions, and Amazon DynamoDB, you can consolidate SSM Patch Manager execution details from multiple accounts, generate comprehensive reports, and distribute them via email and Slack notifications, empowering your team with the insights needed to maintain a secure and compliant infrastructure.

Solution Overview

Figure 1 – Patch reporting for AWS Organizations

The solution leverages several AWS services to automate the patching report generation and delivery process. The solution is deployed using AWS CloudFormation stacks, which provisions the necessary resources, including EventBridge rules, Step Functions, DynamoDB tables, Lambda functions, and required Amazon IAM permissions.

On a scheduled basis, a Lambda function is triggered and retrieves patching data from the DynamoDB tables. These tables store the patching status of each managed node, including details about patch operations such as scan and installs. The lambda function executes a python script that processes this data, generating a comprehensive patching report in a CSV format. This report is then stored in an Amazon Simple Storage Service (Amazon S3) bucket for easy access.

The lambda function also sends the patching operation summaries via Amazon Simple Notification Service (Amazon SNS) or Amazon SES to specified email recipients. If you configured the AWS Chatbot integration with Slack, you can also receive notifications in a Slack channel.

Prerequisites

For this walkthrough, you need to have the following prerequisites:

1. An AWS account.
2. Amazon Elastic Compute Cloud (Amazon EC2) instances and hybrid nodes managed by Systems Manager. To learn more, please see Setting up AWS Systems Manager.
3. Configure patching for above Systems Manager managed nodes.
4. (Optional) Setup Amazon Simple Email Service (SES) and Create an email identity for receiving notifications via SES

Solution Deployment

We are going to share two deployment methods, depending on your use case. The first method is focused on gathering information from all the accounts in AWS Organizations, while the second method is for single account and region.

Method 1: Deploy in AWS Organizations

 For organization-wide deployment, the solution is first deployed in the central (management/delegated administrator(DA)) account, and then a CloudFormation StackSet is used to deploy the necessary resources in member accounts, targeting the entire organization or specific Organizational Units (OUs).

1. CloudFormation stack deployment in central account

1. Download the PatchingReport_Central.yaml CloudFormation template.
2. Navigate to CloudFormation console in AWS account where you would like to deploy the Lambda Function to generate reports
3. For Create Stack, choose with new resources (standard).
4. For Template source, choose Upload a template file. Choose file and select the template you downloaded in step 1.
5. Choose Next
6. For Stack Name, enter the stack name (such as ssm-patch-reporting).
7. In the parameters area, enter the following parameters:
   • For ‘OrganizationID’, specify the Organization ID of the Organization
   • For ‘EventBridgeRuleSchedule’, specify the CRON schedule to run the report generation.
   • For ‘EmailService’, select the service to be used for patch summary email. This can be either SNS or SES.
   • For ‘RecipientEmail’, enter the email address for the report recipient
   • For ‘SenderEmail’, enter the sender email. This parameter is required only when ‘EmailService’ is set to SES. This generates the email in HTML format.
   • For ‘TimeToLive’, enter the number of days to retain the DynamoDB data after it is processed.
   • Set the ‘Chatbot’ to true if you would like to create necessary resources (SNS Topic for Chatbot) to use with AWS Chatbot.
8. In the Configure stack options page, under Capabilities, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Next.
9. In the Review and create page, choose Submit.
10. After the template has deployed, choose Outputs and note the values of the following as shown in Figure 2.
    • PatchInvocationEventBus
    • PatchOperationEventBus
    • ChatbotSNSTopic

Figure 2 – CloudFormation template deployment outputs from Central Account

2. CloudFormation StackSet deployment from the central account

The CloudFormation StackSet deploys the Amazon EventBridge rules for patch invocation and operation in the member accounts of the Organization or specific OUs.

1. Download the PatchingReport_Member.yaml CloudFormation template
2. Navigate to CloudFormation console in AWS management/DA account
3. Select StackSet and choose Create StackSet
4. For Specify template, choose Upload a template file. Choose file and select the template you downloaded in step 1.
5. Choose Next
6. For Stack Name, enter the stack name (such as ssm-patch-reporting-member-acc).
7. In the Parameters area, enter the following parameters in the Member Account(s) section:
   1. For ‘Patch Invocation Event Bus Arn’, enter the PatchInvocationEventBus value you noted from the CloudFormation output in the central account, Step 10 of CloudFormation stack deployment in central account.
   2. For ‘Patch Operation Event Bus Arn, enter the PatchOperationEventBus value you noted from the CloudFormation output in the central account, Step 10 of CloudFormation stack deployment in central account.
8. In the Configure StackSet options page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Next
9. In the Deployment targets,
   1. Choose ‘Deploy to organization’ to deploy to entire organization, or
   2. Choose ‘Deploy to organization units (OUs)’ to deploy to specific organization units.
10. In the Specify regions, choose the required regions. Choose Next
11. In the Review page, choose Submit.

The setup of the centralized patch reporting is now complete and reports are generated regularly as per the schedule mentioned during solution deployment.

Method 2: Single Account – Single Region Deployment

If you don’t have AWS Organizations setup or would like to deploy the solution to a single account and single Region, please follow below steps. Otherwise, skip to Patch operations reporting section.

1. Download the PatchingReport_Single.yaml CloudFormation template.
2. Navigate to CloudFormation console in AWS account
3. For Create Stack, choose with new resources (standard).
4. For Template source, choose Upload a template file. Choose file and select the template you downloaded in step 1.
5. Choose Next
6. For Stack Name, enter the stack name (such as ssm-patch-reporting).
7. In the parameters area, enter the following parameters:
   • For ‘EventBridgeRuleSchedule’, specify the cron schedule to run the report generation.
   • For ‘EmailService’, select the service to be used for patch summary email. This can be either SNS or SES.
   • For ‘RecipientEmail’, enter the email address for the report recipient
   • For ‘SenderEmail’, enter the sender email. This parameter is required only when ‘EmailService’ is set to SES. This generates the email in HTML format.
   • For ‘TimeToLive’, enter the number of days to retain the DynamoDB data after it is processed.
   • Set the ‘Chatbot’ to true if you would like to create necessary resources (SNS Topic for Chatbot) to use with AWS Chatbot.
8. In the Configure stack options page, under Capabilities, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Next.
9. In the Review and create page, choose Submit.

The setup of the patch reporting for single account and single Region is now complete and reports are generated regularly as per the schedule mentioned during solution deployment.

Patch Operations Reporting

During the deployment, you would have chosen between receiving patch operations emails via SNS or SES. These emails contain quick summaries of your patch operations as shown in Figure 3, along with links to detailed reports stored in an S3 bucket. The SNS reports provide plain text summaries, while SES reports are delivered in an HTML tabular format.

Figure 3 – Report notification email from SES

Both report types offer a clear timeline of patching activities, including the total operations performed, successful nodes, failed nodes, and specific command IDs for failures. Conveniently, scan and install operations are separated, allowing you to analyze them independently. If any nodes failed during the patching process, you can use the failed command IDs provided in the report to investigate and troubleshoot the issues.

Figure 4: Detailed Patch reports in CSV format

In addition to the quick summaries, a detailed CSV report is generated and stored in an S3 bucket for your reference as shown in Figure 4. This comprehensive report contains granular information about the patching commands executed across your accounts and Regions. These reports include operation types, targeted instances, operation statuses, and command request timestamps, offering a complete audit trail. The detailed CSV reports are automatically generated and stored in an S3 bucket according to the schedule you specified during the CloudFormation stack deployment, providing you with regular access to the most recent patching data and activities.

AWS Chatbot integration (optional)

If you set the `Chatbot` option to `true` while deploying the solution, it will generate an SNS topic for to be integrated with Chatbot to send patch report notifications in your configured slack channels.

Below are the steps to configure Chatbot for Slack

1. Before you get started, make sure you’ve completed the tasks in Setting up AWS Chatbot.
2. Configure a Slack Client
3. Configure a Slack Channel.
   1. During the slack channel setup, you will have an option to select setup SNS topic to send notifications.
   2. Select the PatchReporting-ChatbotSNSTopic-* that you noted in the Step 10 of Central Account deployment or Single account deployment as shown in Figure 5.
      

      Figure 5 – Chatbot SNS topic

4. You will receive notification about patch report data with S3 URL as shown in Figure 6.
   
   

   Figure 6 – Chatbot Notification in Slack

Clean Up

1. Delete the objects generated in the S3 bucket manually
   1. Open the AWS S3 console and find the S3 bucket generated by CloudFormation
   2. Select all the objects and click Delete
   3. To confirm deletion, type permanently delete in the text input field.
   4. Click ‘Delete Objects’
2. Delete the CloudFormation Stack
   1. Open the AWS CloudFormation console and in the navigation pane, choose Stacks.
   2. Choose the CloudFormation stack that you created earlier, choose Delete, and choose Delete stack.
3. Delete the CloudFormation StackSets
   1. Before you can delete a StackSet, you must first delete all the stack instances within it. Please see Delete stack instances using the AWS Management Console or AWS CLI for detailed steps
   2. Next, Delete a stack set using the AWS Management Console or AWS CLI

Conclusion

By implementing the automated patching report solution, you can streamline the generation and delivery of patching reports across your entire organization. This solution leverages AWS EventBridge, AWS Step Functions, Amazon DynamoDB, AWS Lambda, and AWS CloudFormation to provide a scalable and robust patching reporting mechanism.

With patching data consolidated in a central location and notifications delivered via email and Slack, you gain enhanced visibility into the patching status of your managed nodes. Whether you have a single AWS account or multiple accounts under AWS Organizations, this solution helps you proactively identify and address any patching issues, eliminating manual effort and giving you better control over your patching operations.

To further simplify the troubleshooting process for Systems Manager patching failures, please visit Troubleshooting AWS Systems Manager patching made easy with Amazon Bedrock’s automated recommendations blog post, which offers additional insights and generative AI powered solution to streamline your patching workflow.

About the authors: