Recommended practices for deleting custom inventory types and data using AWS Systems Manager

AWS Systems Manager Inventory provides an easy and secure way to collect metadata and insights from your managed instances. The service provides ready-to-use inventory gatherers for insights on installed applications, network configurations, file metadata, Windows updates, Windows services, Windows server roles/features, and Windows registry. In addition, you can use AWS Systems Manager Inventory to collect any other system property as Inventory, for example rack information on the system or results from a legacy application running on the managed instance.

Recently AWS Systems Manager added capabilities for deleting custom inventory types and deleting data associated with custom inventory types using the delete-inventory AWS CLI or the DeleteInventory API action. This comes along with the ability to get detailed status on the delete operation. The capabilities are integrated with Amazon CloudWatch Events for alarms and AWS CloudTrail for audit logs of the delete operations. Some typical use cases for using these capabilities include getting rid of custom inventory types created by mistake or for testing and cleaning up any Inventory data for certain custom inventory types for a variety of reasons.

In this blog post, I’ll walk you through some things to note about the new capabilities and our recommended practices so you can safely delete custom inventory types or associated data.

Things to note

The capability to delete custom inventory types or data is available using the AWS CLI and API only.
If you have Resource Data Sync set up, deleting inventory data for a custom inventory type doesn’t delete inventory data from your Amazon S3 bucket.
After inventory data is deleted, you will not be able restore the data.
Deleting a custom inventory type does not stop collection of other inventory types in the account.
Deleted or disabled custom inventory types no longer count towards the maximum number of custom types allowed in the account.

Recommended practices

1. Configure Amazon CloudWatch Events to receive notifications for delete operations

CloudWatch Events help you respond to state changes in your AWS resources. You can also create rules that match selected events and route rules to trigger actions. While deleting custom inventory types or data, we trigger events in Amazon CloudWatch for the following conditions:

When inventory data for an instance gets deleted.
When the delete inventory operation is complete.
When one or more instances send inventory data for a custom inventory type that was already disabled.

We recommend that you add a CloudWatch rule in the same Region where you are performing the delete operation for all delete custom inventory operation events. Set a target action, such as sending email notifications. You can add a rule using the AWS Management Console, the AWS CLI or the Amazon CloudWatch Events API. For this blog post we will use the AWS Management Console.

Go to the AWS Management Console and search for the Amazon CloudWatch console.
In the CloudWatch console, on the left navigation pane, choose Events (or Rule if you already have created some Rules).
Choose Create Rule. In Step 1 make the following choices:
1. Choose Event Pattern.
2. For Service name choose EC2 Simple Systems Manager (SSM).
3. For Event Type choose All Events.
On the right side, choose Add Target, then choose SNS Topic. This will show all SNS topics in your AWS account.
Select the SNS topic that is configured to send you an email notification.
Choose Configure details to define a name for the rule.
Provide a name and choose Create Rule to complete the configuration.

Here is an example of a CloudWatch event detail that was triggered at the completion of a delete operation:

{"version":"0",
"id":"c714d01f-b330-6cc4-3004-b82147f5ea0b",
"detail-type":"Inventory Resource State change",
"source":"aws.ssm",
"account":"426243216050",
"time":"2018-04-19T20:17:53Z",
"region":"sa-east-1",
"resources":[],
"detail":{"action-status":"succeeded","action":"delete-summary","resource-type":"managed-instance","resource-id":"","action-reason":"The delete for type name Custom:MyTest123 was completed. The deletion summary is: {"totalCount":0,"remainingCount":0,"summaryItems":[]}","type-name":"Custom:MyTest123"}
}

2. Decide what action you want the system to take if an instance sends inventory data for custom inventory type that was deleted

In case you delete a custom inventory type but your account still has managed instances that might collect and send data for the deleted custom inventory type, you have two options: 1) ignore any new data coming for a deleted inventory type or 2) recreate custom inventory type.

3. First understand what will get deleted

Before you delete a custom inventory type, perform a dry run (without deleting the data). This will help you understand what would get deleted if the operation had been run without using the dry-run option. You can execute the following command to use the dry-run option:

aws ssm delete-inventory --type-name "Custom:custom_type_name" --dry-run

This returns information similar to the following indicating what might have been deleted if the command were run without the dry-run option:

{
   "DeletionSummary":{
      "RemainingCount":3,
      "SummaryItems":[
         {
            "Count":2,
            "RemainingCount":2,
            "Version":"1.0"
         },
         {
            "Count":1,
            "RemainingCount":1,
            "Version":"2.0"
         }
      ],
      "TotalCount":3
   },
   "TypeName":"Custom:custom_type_name"
}

4. Delete custom inventory type or data associated to a custom inventory type

Depending on the decision that you make in Step 2, you can execute the following commands using the AWS CLI for the delete operation to take place.

a) If you want to ignore any new data coming for a deleted inventory type data for the deleted type, execute the following command using the AWS CLI:

aws ssm delete-inventory --type-name "Custom:custom_type_name" --schema-delete-option "DisableSchema"

In this case, the system will return information similar to the following:

{
   "DeletionId":"system_generated_deletion_ID",
   "DeletionSummary":{
      "RemainingCount":3,
      "SummaryItems":[
         {
            "Count":2,
            "RemainingCount":2,
            "Version":"1.0"
         },
         {
            "Count":1,
            "RemainingCount":1,
            "Version":"2.0"
         }
      ],
      "TotalCount":3
   },
   "TypeName":"Custom:custom_type_name"
}

b) If you want to re-create the custom inventory type when a managed instance sends inventory type data for the deleted type, execute the following command using AWS CLI:

aws ssm delete-inventory --type-name "Custom:custom_type_name" --schema-delete-option "DeleteSchema"

In this case the system will return information similar to the following:

{
   "DeletionId":"system_generated_deletion_ID",
   "DeletionSummary":{
      "RemainingCount":3,
      "SummaryItems":[
         {
            "Count":2,
            "RemainingCount":2,
            "Version":"1.0"
         },
         {
            "Count":1,
            "RemainingCount":1,
            "Version":"2.0"
         }
      ],
      "TotalCount":3
   },
   "TypeName":"Custom:custom_type_name"
}

5. Verify that the delete operation is complete

Depending on the amount of inventory data that your instances have collected using AWS Systems Manager, a delete operation could take several minutes to complete. While we do not expect DeleteInventory to cause notable impact to any instance or AWS Systems Manager operations, we recommend that you verify that the delete operation has successfully been completed. Also note that while the actual delete operation might take less time, we check the status every 10 minutes, so it might take up to 10 minutes for the describe-inventory-deletions action to show the status as complete.

You can check the status of a delete operation by using the describe-inventory-deletions CLI command. You can specify a deletion ID to view the status of a specific delete operation. Execute the following command to view the status of a deletion operation. (Note that the system returned the deletion ID in the delete-inventory summary in steps 4a and 4b.)

aws ssm describe-inventory-deletions --deletion-id system_generated_deletion_ID

The system returns the latest status. If the delete operation has not yet finished, wait for few minutes and execute the command again. When a successful delete has been completed, the system returns information similar to the following.

{"InventoryDeletions":
  [
    {"DeletionId": "system_generated_deletion_ID",
     "DeletionStartTime": 1521744844,
     "DeletionSummary":
      {"RemainingCount": 0,
       "SummaryItems":
        [
          {"Count": 1,
           "RemainingCount": 0,
           "Version": "1.0"}
        ],
       "TotalCount": 1},
     "LastStatus": "Complete",
     "LastStatusMessage": "Deletion is successful",
     "LastStatusUpdateTime": 1521745253,
     "TypeName": "Custom:custom_type_name"}
  ]
}

Note that by default the AWS CLI output shows timestamps such as DeletionStartTime and LastStatusUpdateTime in epoch time. To view the AWS CLI output in a friendlier UTC time zone format perform the following steps:

Open your AWS CLI config file. The default location for the AWS CLI Config file is ~/.aws/config. If you don’t find the file at this location, check the value of environment variable AWS_CONFIG_FILE and search in the location.
Reformat the epoch time expression to the ISO8601 UTC time zone by adding or updating (if a value already exists) the value for cli_timestamp_format to iso8601, that is,
```
cli_timestamp_format = iso8601
```
See the AWS CLI Command Reference for more details on AWS CLI general options.

Conclusion

In summary, before you delete a custom Inventory type or delete data associated with a custom inventory type, we recommend that you set up CloudWatch Event rules to get appropriate notifications or actions. Later, before performing the delete, first understand and confirm that you are ok with what the delete operation will delete, and then perform the delete operation. Also, when you are performing a delete operation for a custom type that collects the data from a large number of instances, wait for the delete to be completed before initiating other large-scale operations.

About the Author

Tanu Mutreja is a Sr. Product Manager for AWS Systems Manager. She loves to work on innovative products and solve customer problems with easy solutions and simple experiences.

AWS Cloud Operations Blog