AWS Cloud Operations Blog

Implementing a cross-account and cross-Region AWS Config status dashboard

AWS Config helps central IT administrators monitor the compliance of multiple AWS accounts and multiple regions in large enterprises. AWS Config utilizes a configuration recorder to detect changes in your resource configurations and capture these as configuration items. A separate configuration recorder exists for every region in each AWS account.

However, AWS Config recorders can occasionally stop for various reasons, such as invalid delivery channel permissions, inadvertent error, etc. This Cross-Account Cross-Region AWS Config status dashboard Solution lets you monitor the status of AWS Config recorders across multiple accounts and regions via Amazon CloudWatch custom metrics on a single management account. Then, the administrators can setup alarms on these custom metrics to be notified when the status of these AWS Config recorder changes.

This application is deployed in a management account, and the AWS Lambda functions access each member account and region in an AWS Organization or member accounts list in order to check the AWS Config recorder’s status. It then publishes the status of the AWS Config recorder for each account and region to Amazon CloudWatch as custom metrics. At this point, the operators can define metric alarms on these custom metrics and trigger alarm actions to notify team members via Amazon Simple Notification Service (Amazon SNS).

Architecture

AWS EventBridge invokes an AWS Lambda function called a anager Function based on a scheduled time-based event. Then, the anager function publishes an event to AWS EventBridge for every account in the AWS Organization, or every account set up as part of this solution. This event invokes another Lambda function, called a WorkerFunction. This Worker Function assumes a role in the account for each region and checks for the AWS Config recorder’s status. It then publishes the AWS Config recorder status to Amazon CloudWatch as a custom metric.

Figure 1: Asolution architecture for the cross-account. cross-region Config dashboard

Prerequisites

To utilize our ‘AwsConfigStatusChecker’ application, you need the following tools:

• AWS CLI – Install the AWS CLI
• SAM CLI – Install the SAM CLI. The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications.
• Python 3.8 – Install Python
• At this time, you must launch this solution in the US East (N. Virginia) Region. However, once deployed, the solution will monitor AWS Config recorders for multiple regions and multiple AWS Accounts from the AWS CloudFormation StackSet’s target.

Download the solution

git clone https://github.com/aws-samples/aws-multi-region-multi-account-config-status-dashboard.git

How to build and deploy this application

The application has 2 directories:

• awsconfigstatuscheck-app – A SAM Application with AWS Lambda functions and AWS CloudFormation Template (template.yaml) to deploy the resources by using SAM CLI. This application is deployed in the AWS Organization management account.

• StacksetCFTemplate – This directory contains an AWS CloudFormation StackSet templatefile that defines the IAM Role and IAM Policy deployed on the member accounts. This IAM Policy and IAM Role is necessary on the member accounts for the application to assumeRole and gather AWS Config service status.

  1. Build the application with the sam build command.
awsconfigstatuscheck-app$ sam build

Sample Output:

awsconfigstatuscheck-app$ sam build
Building codeuri: /home/sundjega/TFC/awsconfigstatuschecker/awsconfigstatuscheck-app/aws_config_status_check runtime: python3.8 metadata: {} functions: ['ManagerFunction', 'WorkerFunction']
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
  1. This second command will package and deploy your application to AWS via a series of prompts:
awsconfigstatuscheck-app$ sam deploy --guided --capabilities CAPABILITY_NAMED_IAM

Provide the following as input when you run the command:

    1. Stack Name: Provide any name here for e.g. ‘awsconfigstatuscheck-app’
    2. AWS Region: us-east-1 – This is the supported region for this solution.
    3. Parameters:
      • CheckFrequencyMins – This parameter defines how often the lambda function executes to check for the status of AWS Config s The allowed Values are “rate(5 minutes)”, “rate(15 minutes)”, “rate(30 minutes)”, “rate(60 minutes)”, and “rate(90 minutes)”.
      • CheckAllAccountsinOrgCondition – If set to true, then MemberAccountIds Parameter is ignored, and the application checks the status of AWS Config in all accounts/regions across the AWS Organization. If set to false, then proceed to setting the MemberAccountIds parameter. Allowed Values are “true” or “false”.
      • MemberAccountIds – If the parameter CheckAllAccountsinOrgCondition is set to false, then enter a comma separated list of AWS account IDs where you want the application to check for the status of AWS Config. Leave this parameter to its default value if you set CheckAllAccountsinOrgCondition to true. [e.g., 12345634344,434345555333,455454545324]
    4. Confirm changes before deploy: If set to yes, then any change sets will be shown to you for manual review before execution. If set to no, then the AWS SAM CLI will automatically deploy application changes.
    5. Allow SAM CLI IAM role creation: To deploy this application’s AWS CloudFormation stack, which creates or modifies IAM roles, the CAPABILITY_IAM value for capabilities must be provided. If the permission isn’t provided through this prompt, then in order to deploy this example you must explicitly pass –capabilities CAPABILITY_IAM to the sam deploy command.
    6. Save arguments to samconfig.toml: If set to yes, then your choices will be saved to a configuration file inside the project. This is so that in the future you can just re-run sam deploy without parameters in order to deploy changes to your application.

Sample Output:

awsconfigstatuscheck-app$ sam deploy --guided --capabilities CAPABILITY_NAMED_IAM
Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Not found

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: awsconfigstatuscheck-app
        AWS Region [us-east-1]:
        Parameter CheckFrequencyMins [rate(30 minutes)]: rate(5 minutes)
        Parameter CheckAllAccountsinOrgCondition [true]:
        Parameter MemberAccountIds [IGNORE,IF,CheckAllAccountsinOrg,TRUE]:
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: Y
        Save arguments to configuration file [Y/n]: Y
        SAM configuration file [samconfig.toml]:
        SAM configuration environment [default]:

        Looking for resources needed for deployment: Found!

                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1w2spd13sbzyc
                A different default S3 bucket can be set in samconfig.toml

        Saved arguments to config file
        Running 'sam deploy' for future deployments will use the parameters saved above.
        The above parameters can be changed by modifying samconfig.toml
        Learn more about samconfig.toml syntax at
        https://docs.thinkwithwp.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html


        Deploying with following values
        ===============================
        Stack name                   : awsconfigstatuscheck-app
        Region                       : us-east-1
        Confirm changeset            : True
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-1w2spd13sbzyc
        Capabilities                 : ["CAPABILITY_NAMED_IAM"]
        Parameter overrides          : {"CheckFrequencyMins": "rate(5 minutes)", "CheckAllAccountsinOrgCondition": "true", "MemberAccountIds": "IGNORE,IF,CheckAllAccountsinOrg,TRUE"}
        Signing Profiles             : {}

Initiating deployment
=====================

Waiting for changeset to be created..

CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------
Operation                     LogicalResourceId             ResourceType                  Replacement
---------------------------------------------------------------------------------------------------------------------
+ Add                         AppFunctionPolicy             AWS::IAM::ManagedPolicy       N/A
+ Add                         AppFunctionRole               AWS::IAM::Role                N/A
+ Add                         AssumedFunctionRole           AWS::IAM::Role                N/A
+ Add                         CheckAllAccountsinOrgParame   AWS::SSM::Parameter           N/A
                              ter
+ Add                         ConfigAccountsParameter       AWS::SSM::Parameter           N/A
+ Add                         ManagerFunctionScheduledRul   AWS::Events::Rule             N/A
                              e
+ Add                         ManagerFunction               AWS::Lambda::Function         N/A
+ Add                         PermissionForEventsToInvoke   AWS::Lambda::Permission       N/A
                              Lambda1
+ Add                         PermissionForEventsToInvoke   AWS::Lambda::Permission       N/A
                              Lambda2
+ Add                         WorkerFunctionRule            AWS::Events::Rule             N/A
+ Add                         WorkerFunction                AWS::Lambda::Function         N/A
---------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:changeSet/samcli-deploy1628796085/e50cc209-e848-4bc8-9af4-954872d85a30


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2021-08-12 13:21:41 - Waiting for stack create/update to complete

CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------
ResourceStatus                ResourceType                  LogicalResourceId             ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS            AWS::IAM::ManagedPolicy       AppFunctionPolicy             -
CREATE_IN_PROGRESS            AWS::SSM::Parameter           ConfigAccountsParameter       -
CREATE_IN_PROGRESS            AWS::IAM::ManagedPolicy       AppFunctionPolicy             Resource creation Initiated
CREATE_IN_PROGRESS            AWS::SSM::Parameter           CheckAllAccountsinOrgParame   -
                                                            ter
CREATE_IN_PROGRESS            AWS::SSM::Parameter           CheckAllAccountsinOrgParame   Resource creation Initiated
                                                            ter
CREATE_IN_PROGRESS            AWS::SSM::Parameter           ConfigAccountsParameter       Resource creation Initiated
CREATE_COMPLETE               AWS::SSM::Parameter           CheckAllAccountsinOrgParame   -
                                                            ter
CREATE_COMPLETE               AWS::SSM::Parameter           ConfigAccountsParameter       -
CREATE_COMPLETE               AWS::IAM::ManagedPolicy       AppFunctionPolicy             -
CREATE_IN_PROGRESS            AWS::IAM::Role                AppFunctionRole               -
CREATE_IN_PROGRESS            AWS::IAM::Role                AssumedFunctionRole           -
CREATE_IN_PROGRESS            AWS::IAM::Role                AppFunctionRole               Resource creation Initiated
CREATE_IN_PROGRESS            AWS::IAM::Role                AssumedFunctionRole           Resource creation Initiated
CREATE_COMPLETE               AWS::IAM::Role                AssumedFunctionRole           -
CREATE_COMPLETE               AWS::IAM::Role                AppFunctionRole               -
CREATE_IN_PROGRESS            AWS::Lambda::Function         WorkerFunction                -
CREATE_IN_PROGRESS            AWS::Lambda::Function         ManagerFunction               -
CREATE_IN_PROGRESS            AWS::Lambda::Function         WorkerFunction                Resource creation Initiated
CREATE_IN_PROGRESS            AWS::Lambda::Function         ManagerFunction               Resource creation Initiated
CREATE_COMPLETE               AWS::Lambda::Function         WorkerFunction                -
CREATE_COMPLETE               AWS::Lambda::Function         ManagerFunction               -
CREATE_IN_PROGRESS            AWS::Events::Rule             WorkerFunctionRule            -
CREATE_IN_PROGRESS            AWS::Events::Rule             ManagerFunctionScheduledRul   Resource creation Initiated
                                                            e
CREATE_IN_PROGRESS            AWS::Events::Rule             WorkerFunctionRule            Resource creation Initiated
CREATE_IN_PROGRESS            AWS::Events::Rule             ManagerFunctionScheduledRul   -
                                                            e
CREATE_COMPLETE               AWS::Events::Rule             ManagerFunctionScheduledRul   -
                                                            e
CREATE_COMPLETE               AWS::Events::Rule             WorkerFunctionRule            -
CREATE_IN_PROGRESS            AWS::Lambda::Permission       PermissionForEventsToInvoke   -
                                                            Lambda2
CREATE_IN_PROGRESS            AWS::Lambda::Permission       PermissionForEventsToInvoke   -
                                                            Lambda1
CREATE_IN_PROGRESS            AWS::Lambda::Permission       PermissionForEventsToInvoke   Resource creation Initiated
                                                            Lambda2
CREATE_IN_PROGRESS            AWS::Lambda::Permission       PermissionForEventsToInvoke   Resource creation Initiated
                                                            Lambda1
CREATE_COMPLETE               AWS::Lambda::Permission       PermissionForEventsToInvoke   -
                                                            Lambda2
CREATE_COMPLETE               AWS::Lambda::Permission       PermissionForEventsToInvoke   -
                                                            Lambda1
CREATE_COMPLETE               AWS::CloudFormation::Stack    awsconfigstatuscheck-app      -
---------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
---------------------------------------------------------------------------------------------------------------------
Outputs
---------------------------------------------------------------------------------------------------------------------
Key                 LambdaFunction1
Description         ManagerFunction ARN
Value               arn:aws:lambda:us-east-1:xxxxxxxxxx:function:ManagerFunction

Key                 LambdaFunction2
Description         WorkerFunction ARN
Value               arn:aws:lambda:us-east-1:xxxxxxxxxx:function:WorkerFunction
---------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - awsconfigstatuscheck-app in us-east-1
  1. Create an AWS CloudFormation StackSet by using AWS CLI. Provide the AWS Management account ID where this StackSet will be deployed.
StacksetCFTemplate$ aws cloudformation create-stack-set \ 
    --description "role/policies for aws config checker" \ 
    --stack-set-name aws-config-checker-stackset \ 
    --template-body file://stacksettemplate.yml \ 
    --permission-model SERVICE_MANAGED \ 
    --capabilities CAPABILITY_NAMED_IAM \ 
    --parameters ParameterKey=AdministratorAccountId,ParameterValue= \ 
    --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false

Sample Output:


StacksetCFTemplate$ aws cloudformation create-stack-set \
> --description "role/policies for aws config checker" \
> --stack-set-name aws-config-checker-stackset \
> --template-body file://stacksettemplate.yml \
> --permission-model SERVICE_MANAGED \
> --capabilities CAPABILITY_NAMED_IAM \
> --parameters ParameterKey=AdministratorAccountId,ParameterValue=xxxxxxxxxx \
> --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false
{
"StackSetId": "aws-config-checker-stackset:273c07d1-c7ab-4e6c-b388-0f6e2263c400"
}

  1. Create an AWS CloudFormation Stack Instances by using AWS CLI. Provide the OrganizationalUnitIds of the member accounts to which the Stack Instances are deployed.
StacksetCFTemplate$ aws cloudformation create-stack-instances \ 
    --deployment-targets OrganizationalUnitIds= \ 
    --stack-set-name aws-config-checker-stackset --regions us-east-1

Sample Output:

StacksetCFTemplate$ aws cloudformation create-stack-instances \
>     --deployment-targets OrganizationalUnitIds=r-xxxx \
>     --stack-set-name aws-config-checker-stackset  --regions us-east-1
{
    "OperationId": "1e740a93-e980-44e0-a4d6-6747384dab2f"
}

Troubleshooting the Lambda function

In order to simplify troubleshooting, SAM CLI has a command called sam logs. sam logs lets you fetch logs generated by your deployed Lambda function from the command line. In addition to printing the logs on the terminal, this command has several useful features to help you quickly find the bug.

awsconfigstatuscheck-app$ sam logs -n WorkerFunction --stack-name awsconfigstatuscheck-app --tail

awsconfigstatuscheck-app$ sam logs -n ManagerFunction --stack-name awsconfigstatuscheck-app --tail

View the status of AWS Config Recorders across target accounts as Amazon CloudWatch Custom Metrics.

After a few minutes, log in to Amazon CloudWatch Console to view custom metrics of AWS Config recorders by region and account.

This is a screenshot of the Amazon CloudWatch highlighting the custom metric called AWSConfigStatus that is published as part of this solution.

Figure 2: Sample Amazon CloudWatch Console

As shown below, you can create a dashboard in Amazon CloudWatch in order to view the AWSConfig Status custom metrics in a single view. Create alarms in CloudWatch in order to watch for a decrease in the number of running AWS Config recorders. Then, define an action in order to send a notification to an Amazon SNS Topic and/or create an OpsItem or incident in AWS Systems Manager.

Screenshot of a sample AWS Cloudwatch dashboard showing how a single dashboard can be utilized to visualize the AWS Config recorder status across multiple accounts and regions.

Figure 3: AWS CloudWatch Dashboard Sample

Cleanup

  1. To delete the AwsConfigStatusCheck application that you created, use the SAM CLI.
awsconfigstatuscheck-app$ sam delete
Are you sure you want to delete the stack awsconfigstatuschecker-app in the region us-east-1 ? [y/N]: y
Are you sure you want to delete the folder awsconfigstatuschecker-app in S3 which contains the artifacts? [y/N]: y
- Deleting S3 object with key awsconfigstatuschecker-app/aca923a2899ecef9bb25fc9d708b2067
- Could not find and delete the S3 object with the key awsconfigstatuschecker-app/aca923a2899ecef9bb25fc9d708b2067
- Deleting S3 object with key awsconfigstatuschecker-app/23b0a7ffd14c825f099b0d67a2d1ae18.template
- Deleting S3 object with key awsconfigstatuschecker-app/26d62b2b9477b02ff57126cd6a59be0b.template
- Deleting S3 object with key awsconfigstatuschecker-app/938f74b20506773321240e7555ac4910.template
- Deleting S3 object with key awsconfigstatuschecker-app/93c76ece26e084135707d1b4b26f053b.template
- Deleting S3 object with key awsconfigstatuschecker-app/b93be3ce737bcac83d43d92246689245.template
- Deleting S3 object with key awsconfigstatuschecker-app/f81beb0c1361619d6542ef6017556482.template
- Deleting Cloudformation stack awsconfigstatuschecker-app
Deleted successfully
  1. Delete the Stack Instances from AWS CloudFormation StackSet using AWS CLI.
StacksetCFTemplate$ aws cloudformation delete-stack-instances --stack-set-name aws-config-checker-stackset --regions us-east-1 --no-retain-stacks --deployment-targets OrganizationalUnitIds=
  1. Wait for Step 2 to finish. Then, delete the AWS CloudFormation StackSet using AWS CLI.
StacksetCFTemplate$ aws cloudformation delete-stack-set --stack-set-name aws-config-checker-stackset

Conclusion

This post provided a solution that can be easily deployed on a management account in order to monitor the status of AWS Config recorders on multiple accounts and regions. The solution publishes custom metrics to Amazon CloudWatch. AWS administrator and operators can setup CloudWatch alarms to be notified when the status of an AWS Config recorder’s status changes for any reason .

About the authors

Image of Author Jegan Sundarapandian

Jegan Sundarapandian

Jegan Sundarapandian is a Sr. Technical Account Manager with AWS. He works with AWS Customers to implement AWS best practices and keep them operationally healthy.

About the author Venugopalan Vasudevan

Venugopalan Vasudevan

Venugopalan Vasudevan is a Senior Technical Account Manager at AWS who is based in Denver, Colorado. Venu works with AWS customers to solve architectural, operational, and cost optimization challenges and help them build on AWS. In his spare time, he enjoys playing with his two kids.