AWS Cloud Operations Blog

Understanding the differences between configuration history and configuration snapshot files in AWS Config

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.


When you run your applications on AWS, you often use AWS resources, which you must create and manage collectively. As the demand for your application keeps growing, so does your need to keep track of your AWS resources. AWS Config tracks changes made to these supported AWS resources and records their changes as configuration items (CIs), which are JSON files delivered to an Amazon Simple Storage Service (Amazon S3) bucket. These files are accessed through AWS Config APIs and optionally sent through Amazon Simple Notification Service (Amazon SNS).

AWS Config delivers three types of configuration files to the S3 bucket:

In this blog post, we discuss the difference between configuration history and configuration snapshot files and the use cases for which they qualify.

Configuration history file

The JSON formatted configuration changes for a resource called the CIs are delivered as configuration history files to a S3 bucket every 6 hours.

Figure 1: Configuration history file delivery

  1. An AWS resource undergoes configuration changes.
  2. The configuration recorder captures those changes.
  3. Configuration items (CIs) are generated in JSON format.
  4. The CIs are delivered as configuration history files to a designated S3 bucket every 6 hours.

 

Example:

123456789012_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20201007T123229Z_20201007T123229Z_1.json.gz

Configuration snapshot file

A JSON formatted point-in-time capture of the configurations of all the resources are delivered to the S3 bucket as Snapshot file

Figure 2: Configuration snapshot file delivery

  1. A user manually performs the DeliverConfigSnapshot API call or the API is invoked according to a periodic setting for the snapshot in the configuration recorder.
  2. AWS Config performs a point-in-time capture of the configurations of all the resources.
  3. The CIs are generated in JSON format.
  4. The CIs are delivered as configuration snapshot files to a designated S3 bucket.

The following table summarizes the differences between the two files.

  Configuration history Configuration snapshot
Definition Collection of the CIs for a specific supported resource type over any time period. Collection of the CIs for all supported resources that exist in an account. It is a point-in-time capture of all the resource configurations.
Format JSON JSON
Action required

After you enable the configuration recorder, history files are enabled automatically.

No action required.

You must enable snapshot files manually.
How to enable

Enabled when you set up the configuration recorder.

No action required.

Use the AWS CLI to enable.
Delivery frequency Delivery to an S3 bucket every 6 hours. Delivery to an S3 bucket can be set to every 1, 3, 6, 12 or 24 hours.
When are files delivered Only if a change occurs on a resource. On a defined frequency or invoked on demand.
File delivery status

Command:

aws configservice describe-delivery-channel-status

Check the configHistoryDeliveryInfo key              in the output.

Example:

"configHistoryDeliveryInfo": 
{
  "lastSuccessfulTime": 1450317838.412,
  "lastStatus": "SUCCESS",
  "lastAttemptTime": 1450317838.412
}

Command:

aws configservice describe-delivery-channel-status

Check  the configSnapshotDeliveryInfo key in the output.

Example:

"configSnapshotDeliveryInfo": 
{
  "lastSuccessfulTime": 1452185597.094,
  "lastStatus": "SUCCESS",
  "lastAttemptTime": 1452185597.094
}
Naming convention

<Account_ID>_Config_<Region>_Config History_AWS<Resource_Type>_<Timestamp>.json.gz

Example:

123456789012_Config_eu-west-1
_ConfigHistory_AWS::EC2::SecurityGroup_
20180816T155344Z_20180816T155350Z_1.json.gz

<Account_ID>_Config_<Region>_ConfigSnapshot_ <Timestamp>_<Snapshot_ID>.json.gz

Note: Each snapshot file has a unique snapshot ID.

Example:

123456789012_Config_eu-west-1
_ConfigSnapshot_20180816T155344Z
_4f0d4257-61fe-46a3-9573-2afe04d3c8e9.json.gz
Related SNS notifications
Note: You can configure AWS Config to stream configuration changes and notifications to an Amazon SNS topic.
Use cases

When you’re looking for the historical configuration state for a specific resource type.

For example:

  • List the number of changes (CIs) that happened to an EBS volume.
  • Retrieve the number of CIs recorded per month by AWS Config.

When you’re looking for the current configuration state for all supported resources in an account.

For example:

  • What is the latest configuration state of all resources that currently exist in my account?
  • Retrieve the list of all resources recorded by AWS Config in a specific Region only.

Note: The S3 bucket also contains an empty file named ConfigWritabilityCheckFile. AWS Config creates this file to verify that the service has permissions to successfully write to the S3 bucket.

Popular use cases

Because AWS Config delivers Configuration history and snapshot files to the S3 bucket, you can use the service’s integration with Amazon Athena to query either file type.

You can also customize your query and use Amazon QuickSight to visualize the data according to your use case. For more information, see the Visualizing AWS Config data using Amazon Athena and Amazon QuickSight blog post.

FAQs

Q. Can I use the console to enable delivery of configuration snapshot files?

No. Currently, you must use the AWS CLI. Use the PutDeliveryChannel API to create a channel to deliver configuration information and then configure the delivery frequency.

Q. Can I have snapshot files delivered on demand or is the delivery only periodic?

Yes. Use the deliver-config-snapshot command to deliver the snapshot file on demand.

Q. Do configuration history and configuration snapshot record CI in the same way?

Yes. The configuration recorder records all changes in the Region and delivers the same CI in the same format to the S3 bucket in two different folders (ConfigHistory and ConfigSnapshot). 

Q. How can I view these files?

You can use the Amazon S3 console and the AWS CLI to view the files.

If you use the console:

There are two folders (ConfigHistory and ConfigSnapshot) in the following path:

config-bucket-XXXXXXXX/AWSLogs/<AccountID>/Config/<Region>/YYYY/MM/DD

On the bucket page in the S3 console, there are ConfigHistory and ConfigSnapshot folders.

Figure 3: Configuration history and snapshot files in the Amazon S3 console

 

ConfigHistory

On the bucket page in the S3 console, there are two configuration history files, both of which were last modified on September 10, 2019.

Figure 4: Configuration history files in the Amazon S3 console

ConfigSnapshot

On the bucket page of the S3 console, there is one configuration snapshot file. It was last modified on September 10, 2019.

Figure 5: Configuration snapshot file in the Amazon S3 console

If you’re using the AWS CLI:

To retrieve configuration history files, use the get-resource-config-history command and specify the resource type and the resource ID.

Note: You cannot use the AWS CLI to retrieve configuration snapshot files.

Command

$ aws configservice get-resource-config-history --resource-type AWS::EC2::SecurityGroup --resource-id sg-123456ab

Q. If the files in the S3 bucket are deleted, will it affect the timeline view in the AWS Config console?

No. The configuration history and snapshot files delivered to an S3 bucket are for your auditing and analysis only. The configuration information displayed in the AWS Config console is from its data store.

Q. Does the retention period in the configuration recorder setting affect S3 files?

No. This retention period does not apply to the files in the S3 bucket. The files in the AWS Config data store depend on the retention period setting in the Config. By default, this value is 7 years. 

Q. If I’m using an aggregator, are the CIs of member accounts delivered to the master account?

Only current state CIs that represent the configuration and compliance state of AWS resources can be aggregated into an AWS Config aggregator. Historical state CIs of member accounts are delivered in the history files for each account. You can designate a central S3 bucket to deliver the history files of all member accounts into a central bucket.

Q. Can I use advanced querying to query the history and snapshot files?

No. Advanced querying can only be used to query the current configuration and compliance state of your resources natively in the console or through APIs. You can use Amazon Athena to query the history and snapshot files in your S3 bucket.

Conclusion

In this post, we showed you the differences between configuration history and configuration snapshot files. With this information, you can now start monitoring your account for recorded changes for billing and other reasons required by your use case. You can take advantage of the AWS Config integration with Amazon Athena, Amazon ES, and Kibana to extract the details in these files.

About the authors

Attalla is a Sr. Cloud Support Engineer. He specializes in AWS Config, Logging and Monitoring AWS services. Attalla is passionate about building secure and effective solutions for AWS customers. He enjoys deep dive into compliance related challenges, writing articles and building tutorials. Outside of work, Attalla’s favorite activities include snorkeling and reading. He loves spending time with his daughters.

Sagara is a Cloud Support Engineer. She has a passion for providing customers with technical guidance of AWS Security services and coming up with innovative solutions to customer issues. Outside of work, she enjoys spending time with her family and going on hikes.

Sushma is Cloud Support Engineer at AWS. She is a multi-domain enthusiast with a vast experience in the field of cyber-security and is passionate about coding solutions to automate processes. She is motivated to innovate ideas which simplifies solutions and develop articles with gotchas for the customers. Outside of work she is a painter, dancer, enjoys a good Netflix binge but can also be found on long bike rides on hilly country roads.