AWS Cloud Operations Blog

How to get a daily report for your resources configuration changes

AWS allows customers to build, experience, and innovate in their AWS accounts, resulting in dynamic environments.

You can manage your resources changes using different controls, such as:

In addition to the existing controls, Security, Governance, and Operations teams might want additional visibility over the resource configuration changes. AWS Config  lets you continuously monitor and record your AWS configurations for supported services. Using advanced queries, we can run queries against our resources inventory.

In this post, you’ll learn how to generate a scheduled resource configuration report. The report will include newly-added resources and resource configuration changes across your multi-account and multi-region AWS environment. Governance teams can review the report, identify risks, track the latest changes, and gain visibility over the environment.

We’ll also share code that will provision a daily AWS Config query, with a CSV report generator and email sender. As a result, the recipients can receive a detailed scheduled report.

Solution overview

All of the samples from the post can be found in this Github repository.

The solution includes a serverless architecture for running a query against AWS Config. Amazon EventBridge Rule will trigger an AWS Lambda function every day in a specific configurable time.

The following diagram describes the AWS Config Daily Reporter flow:

Figure 1. Architecture diagram for AWS Config Daily Reporter

Figure 1. AWS Config Daily Reporter

  1. The Lambda function will run a query against AWS Config, thereby getting the changed configuration during the current day.
  2. The Lambda function will generate a CSV report and send it via email using Amazon Simple Email Service (Amazon SES).

Prerequisites

Before getting started, make sure that you have a basic understanding of the following:

You will also need to have a pre-configured Multi-Account AWS Config Aggregator and Amazon SES for sending email.

Solution Walkthrough

The Python code in this post was written using the AWS Cloud Development Kit (AWS CDK). To view the code, see the associated GitHub repository. If you’re unfamiliar with AWS CDK, then see Getting started with AWS CDK.

For the deployment, use the AWS CDK stack cdk/cdk/cdk_stack.py. This stack requires the following parameters:

  • aggregator – Name of AWS Config Aggregator.
  • sesarn – The Amazon SES arn.
  • RECIPIENT – Email recipient that will get the CSV report.
  • SENDER – Email sender as configured on Amazon SES.
  • HOUR – The hour (UTC) that the Lambda will run.
  • MINUTE – The minute (UTC) that the Lambda will run.

AWS Config Aggregator

An aggregator is an AWS Config resource type that collects AWS Config configuration data from multiple accounts and regions.

Our solution will use an aggregator to run queries against the environment.

Set Up an Aggregator Using the Console and note your AWS Config aggregator name.

Amazon SES

Amazon SES is a cost-effective, flexible, and scalable email service that enables developers to send mail from within any application.

In our solution, we will use Amazon SES to send the configuration changes report CSV via Email.

Setup Amazon Simple Email Service and make a note of your email sender and the Amazon SES arn.

Deploying

  1. Make sure you are logged in to the AWS management console, and have configured your AWS CLI credentials. More information can be found here.
  2. Clone the solution repository
git clone https://github.com/aws-samples/config-daily-report
  1. Navigate to the cdk directory of the cloned repository:
cd config-daily-reporter/cdk
  1. Run cdk bootstrap:
cdk bootstrap
  1. Deploy the solution:
cdk deploy
--parameters aggregator=<aggregator name> \
--parameters sesarn=<ses arn> \
--parameters RECIPIENT=<recipient email address> \
--parameters SENDER=<sender email address>
--parameters HOUR=<time in UTC (hour)>
--parameters MINUTE==< time in UTC (minute)>

The “cdk deploy” command adds a new AWS CloudFormation template, which creates a Lambda Function with a daily trigger. At the end of the deployment, the Lambda function will be triggered automatically, and you should get an email a few minutes later.

Review the code

The config_reporter Lambda function is written in Python 3.8. In this section, we’ll go over the code which can be found here.

The create_report function will run a query against the configured AWS Config Aggregator for any changes made during the last day:

def create_report(aggregator_name, today):
    client = boto3.client('config')
    response = client.select_aggregate_resource_config(
        Expression=f"SELECT * WHERE configurationItemCaptureTime LIKE '{today}%'",
        ConfigurationAggregatorName=aggregator_name
    )
    changed_resources = response["Results"]
    json_list = [json.loads(line) for line in changed_resources]

For example, for June 09 2022, it will run the following query:

SELECT * WHERE configurationItemCaptureTime LIKE '2022-06-09%'

Next, we’ll transform the JSON results into a CSV file.

    for resource in json_list:
        AWS_REGION = resource['awsRegion']
        RESOURCE_ID = resource['resourceId']
        RESOURCE_TYPE = resource['resourceType']
        resource['Link'] = get_link(AWS_REGION, RESOURCE_ID, RESOURCE_TYPE)      
        print (resource)
    all_fields = set()
    for item in json_list:
        all_fields.update(item.keys())
    # Save the report file
    with open(filename, 'w', newline='') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=list(all_fields))
        writer.writeheader()
        writer.writerows(json_list)
    print("Report generated "+ filename)

The CSV will be saved into the /tmp folder.

For each resource configuration change, we’ll add a column with a link to the resource in the AWS Config console. The get_link function will be used to generate the link:

def get_link(AWS_REGION, RESOURCE_ID, RESOURCE_TYPE):
    url = f'https://{AWS_REGION}.console.thinkwithwp.com/config/home?region={AWS_REGION}#/resources/timeline?resourceId={RESOURCE_ID}&resourceType={RESOURCE_TYPE}'
    return url

Eventually, we’ll use the send_email function to send an email using Amazon SES, with the CSV report as an attachment:

def send_email(today, SENDER, RECIPIENT, filename):
    # The subject line for the email.
    SUBJECT = f"AWS Config changes report {today}"
    ATTACHMENT = filename
    BODY_TEXT = "Hello,\r\nPlease see the attached file which includes the changes made during the last day."
    ses = boto3.client('ses')

    # The HTML body of the email.
    BODY_HTML = """\
    <html>
    <head></head>
    <body>
    <p>Hello, please see the attached file which includes the changes made during the last day.</p>
    </body>
    </html>
    """
    CHARSET = "utf-8"
    msg = MIMEMultipart('mixed')
    msg['Subject'] = SUBJECT
    msg['From'] = SENDER
    msg['To'] = RECIPIENT
    msg_body = MIMEMultipart('alternative')
    textpart = MIMEText(BODY_TEXT.encode(CHARSET), 'plain', CHARSET)
    htmlpart = MIMEText(BODY_HTML.encode(CHARSET), 'html', CHARSET)
    msg_body.attach(textpart)
    msg_body.attach(htmlpart)
    att = MIMEApplication(open(ATTACHMENT, 'rb').read())
    att.add_header('Content-Disposition', 'attachment',
                   filename=os.path.basename(ATTACHMENT))
    msg.attach(msg_body)
    msg.attach(att)
    # Provide the contents of the email.
    response = ses.send_raw_email(
        Source=SENDER,
        Destinations=[
            RECIPIENT
        ],
        RawMessage={
            'Data': msg.as_string(),
        }
    )
    print("Email sent! Message ID:"),
    print(response['MessageId'])

            RawMessage={
                'Data': msg.as_string(),
            }
        )
    # Display an error if something goes wrong.
    except ClientError as e:
        print(e.response['Error']['Message'])
    else:
        print("Email sent! Message ID:"),
        print(response['MessageId'])

Results

As a result, we should get the CSV report with the last day configuration changes to the recipient mail box.

The following table represents an example of the report with one changed Amazon Elastic Compute Cloud (Amazon EC2) instance:

Figure 2. AWS Config sample resources configuration changes report

Figure 2. AWS Config sample resources configuration changes report

Selecting the link of the resource will take you to the resource timeline in the AWS Config console:

Figure 2. AWS Config Resource Timeline Sample

Figure 3. AWS Config Resource Timeline Sample

Cleanup

To avoid incurring future charges, delete all the resources used in this solution. Use the following steps to clean up the resources:

In order to delete the solution, please follow this steps:

  1. Delete the AWS Config Aggregator.
  2. Delete the Amazon SES.
  3. Use cdk cli with the cdk destroy command
Are you sure you want to delete: cdk-stack (y/n)? y
cdk-stack: destroying...
....
DELETION OF RESOURCES
....

 ✅  cdk-stack: destroyed

Conclusion

In this post, we demonstrated how to run a scheduled advanced query against AWS Config, and generate daily configuration report changes.

Use this solution in order to get visibility over your daily changes in your multi-account and multi-region AWS environment.

You can also modify code to run the solution in a different frequency, or to run different queries. For example: query for specific resources.

About the author:

Daniel Begimher

Daniel is a senior security consultant who works with customers to help protect their cloud resources and data. He works with customers to mitigate and improve security by working backwards, getting to know the customer, and spotting potential vulnerabilities. Outside of work, you may find him playing with a new gadget, enjoying a video game, or traveling to explore new places and flavors.