Use AWS Lambda with AWS Control Tower Audit account to inspect your multi-account setup

When you are building workloads on AWS, you are encouraged to follow a multi-account strategy to isolate workloads into multiple AWS accounts. You can do this to separate your accounts based on different business units, different stages of the software development lifecycle (SDLC) or another manner that is suitable for your organization’s needs. Whichever approach you decide to take, a multi-account strategy provides you the right level of workload isolation and separation of concerns in your AWS environment. With decades of experience in architecting various workloads, we introduced AWS Control Tower as a service that provisions a managed landing zone. A landing zone refers to a well-architected multi-account AWS environment. Control Tower allows you to manage and govern your multi-account setup at scale. As your AWS environment grows, you will often need to inspect resources across your multiple AWS accounts for different needs. In this post, we will show you how you can inspect your multi-account AWS environment with the help of AWS Lambda.

Control Tower Foundational accounts

When you set up Control Tower, you start with three foundational AWS accounts:

Management account – This account serves as the primary payer account for your Control Tower-based environment. You will use this account to manage your AWS accounts, organizational units (OUs) in your AWS Organizations and to check the compliance of your AWS environment with the help of guardrails established for security and compliance. You should avoid running any workloads in this account.
Log Archive account – You will use the Log Archive account to centrally store and analyze logs generated for API activities by AWS CloudTrail from your different AWS accounts. You may also use this account for a centralized logging solution for logs generated by your workload accounts, using Kinesis Firehose, Amazon OpenSearch Service or other options.
Audit account – This account is used to provide your security and compliance teams, read and write access to all accounts in your Control Tower environment. It enables this by providing programmatic access through Lambda functions. We will look at Audit account in detail in the next section, as this is central to the topic we are addressing in this post.

Control Tower Audit account and cross-account access

The Audit account provides the ability to programmatically obtain read/write access to other accounts governed by Control Tower using Lambda functions. You may also choose to use this account as the delegated admin for security services such as AWS Security Hub, Amazon GuardDuty, Amazon Macie, and others. This gives you the ability to centralize your security and compliance needs. Additionally, it gives you access to AWS Organizations API, which can be useful when you want to inspect other accounts, as we will see later.

There may be reasons why you want to have a separate “Security account” which serves as the centralized delegated admin for your security services. Neither approach is preferred over the other and they both work equally well. The choice you make usually depends on how your security and compliance departments are organized.

To enable access to other accounts, Control Tower creates two AWS Identity and Access Management (IAM) roles in the Audit account:

aws-controltower-AuditAdministratorRole

Permissions –
- AWSLambdaExecute – AWS Managed Policy required for Lambda execution
- AssumeRole-aws-controltower-AuditAdministratorRole – Custom inline policy to only allow this role to assume aws-controltower-AdministratorExecutionRole in other accounts
Trust relationship – lambda.amazonaws.com

aws-controltower-AuditReadOnlyRole

Permissions –
- AWSLambdaExecute – AWS Managed Policy required for Lambda execution
- AssumeRole-aws-controltower-AuditReadOnlyRole – Custom inline policy to only allow this role to assume aws-controltower-ReadOnlyExecutionRole in other accounts
Trust relationship – lambda.amazonaws.com

Architecture diagram showing Lambda functions in the Control Tower Audit account assuming roles in other workload accounts to perform cross-account inspection.

Figure 1. Architectural diagram showing how Control Tower Audit account can access other accounts

Figure 1 above shows how the Audit-related roles in the Audit account can be used to access other accounts. This is enabled by the these corresponding IAM roles in each of the accounts governed by Control Tower:

aws-controltower-AdministratorExecutionRole

Permissions –
- AdministratorAccess – AWS Managed Policy granting full administrator access in this account
Trust relationship – arn:aws:iam::<AuditAccount>:role/aws-controltower-AuditAdministratorRole

aws-controltower-ReadOnlyExecutionRole

Permissions –
- ReadOnlyAccess – AWS Managed Policy granting full administrator access in this account
Trust relationship – arn:aws:iam::<AuditAccount>:role/aws-controltower-AuditReadOnlyRole

These roles within the individual accounts have a trust relationship with the Audit-focused IAM roles in the Control Tower Audit account. This enables a Lambda function in the Audit account to use either of the two Audit IAM roles as the execution role. The Lambda can then assume the corresponding role in the individual accounts based on the trust relationship. We will see this in action when we look at an example use case in the next section.

Note: The two IAM roles in the individual workload accounts do not exist in the Control Tower Management account. This is because the Audit account is intended to be used by the security and compliance teams. These teams do not always need access to the Management account, which is the most powerful account in your Control Tower environment and contains sensitive information about governance and billing. Additionally, since your Management account is not expected to have any workloads, you shouldn’t have to inspect your Management account from your Audit account.

Example use case – Find Dangling DNS records

Let us now look at how this mechanism could be used to identify public Amazon Route 53 DNS records that reference IP addresses which are no longer owned by accounts under your AWS Organizations.

This scenario, commonly referred to as Dangling DNS, occurs when a DNS record previously created, either in Route53 or any other DNS service, is left dangling when the underlying IP address is no longer owned by you and available for other users in the public IP pool. This scenario commonly impacts subdomains created to host non-production stacks, which may not be as strictly governed as the production stack domains. Attackers can perform a brute force effort against these common subdomains which point to an IP address and try to procure that IP by launching new Amazon EC2 instances, new EIPs or other means of acquiring a public IP. While the attacker cannot request for, nor is guaranteed to be assigned this IP address, there is a possibility they randomly get ownership of this IP. With the subdomain still pointed to this IP address, the attacker now has access to all traffic initiated against that subdomain.

Prerequisites

AWS Organizations API is by default only accessible from your Management account. You may, however, designate your Audit account as the delegated admin for one of the services which supports delegated administrator. Doing this allows you to make use of the Organizations API in the Audit account to fetch the all accounts in your organization. You can then iterate over your accounts to perform your inspection. You can execute the below command in the AWS Command Line Interface (CLI) as an administrator in your Management account to designate the Audit account as the delegated admin.

aws organizations register-delegated-administrator \
--account-id <Your-Audit-Account-Id> \
--service-principal account.amazonaws.com

With the Audit account as the delegated admin, you can configure the aws-controltower-AuditReadOnlyRole role with least privilege permissions to list accounts. The Audit roles in Control Tower are governed by preventive guardrails and can only be modified by the AWSControlTowerExecution role in the Management account. With the AWSControlTowerExecution role assumed by a principal in the Management account, you can attach the permissions to list accounts to the aws-controltower-AuditReadOnlyRole as shown below.

"Statement": [
        {
            "Sid": "org-account-policies",
            "Effect": "Allow",
            "Action": "organizations:ListAccounts",
            "Resource": "*"
        }
]

Note: If you are unable to set up the Audit account as a delegated admin, you can alternatively use AWS Systems Manager Parameter Store or an Amazon S3 bucket in the Audit account, where you can programmatically update the accounts as they are enrolled/deregistered from the Control Tower using the AWS Control Tower Lifecycle Events (CreateManagedAccount and UpdateManagedAccount events). Your Audit role will need additional permissions to access the service you decide to use.

With the prerequisites in place, here are the steps to demonstrate how you can identify such dangling DNS records in your AWS environment. The complete code and automation to setup this Lambda function in your Control Tower Audit account using AWS Serverless Application Model is available in this GitHub project.

Step 1. Fetch and Iterate over your Control Tower Accounts and assume the aws-controltower-* role

We will first fetch all the accounts in your AWS Organizations and iterate over them to assume the aws-controltower* role within each account using the AWS Security Token Service (STS). In this example we want to just get a list of Dangling DNS records, so we assume the role with read capabilities i.e., aws-controltower-ReadOnlyExecutionRole role; if your use case requires to create/update/delete any resource from the account, assume the aws-controltower-AuditAdministratorRole instead.

client = boto3.client('organizations')
response = client.list_accounts()
sts_connection = boto3.client('sts')
acct_b = sts_connection.assume_role(
RoleArn="arn:aws:iam::<AccountId>:role/aws-controltower-ReadOnlyExecutionRole",
RoleSessionName="cross_acct_lambda"
)
ACCESS_KEY = acct_b['Credentials']['AccessKeyId']
SECRET_KEY = acct_b['Credentials']['SecretAccessKey']
SESSION_TOKEN = acct_b['Credentials']['SessionToken']

In iterating over the accounts, you will need to skip the Management account, as it does not have the Control Tower Audit roles as described before. Depending on your use case, you may choose to exclude Audit and Log Archive accounts also.

Step 2. Fetch hosted zones from Route 53

In this step, we fetch hosted zones from Amazon Route 53.

client = boto3.client(
        'route53',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
        aws_session_token=SESSION_TOKEN,
)
response = client.list_hosted_zones()

Route53 supports Private hosted zones, which are only resolvable from within a VPC. As a best practice, any internal-only DNS entries should be setup under a private hosted zone. As we are interested only in publicly resolvable DNS records, Private hosted zones are filtered out from the next step.

Step 3: Fetch A records from the hosted zone

In this step, we will fetch resource records within a hosted zone from the previous step.

response = client.list_resource_record_sets(
HostedZoneId='<HostedZoneID>'
)

Route 53 supports various types of DNS records like CNAME, PTR, TXT, and others. We are interested only in non-aliased A record types which is used to associate IP addresses to a DNS Records. All other record types will be filtered out from the next step.

Step 4: Identify if you own the EIP

A common pattern to manage DNS entries for public facing workloads hosted on EC2 instances is by mapping domains to Elastic IPs. Elastic IPs provide static IPv4 addresses that can be associated with EC2 instances so that any changes to public IP addresses on the EC2 instance can be made seamless to the DNS record and the external world. One approach of identifying if the EIP is still owned by the account is to scan the CloudTrail logs to see if there was a DisassociateAddress API call made for the EIP. Alternatively, as described in the snippet below, we can use the EC2 API to describe the IP address – an error in the response would indicate that the EIP is not currently owned by the account.

client = boto3.client(
        'ec2',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
        aws_session_token=SESSION_TOKEN,
)
response = client.describe_addresses(
        PublicIps=['<IP Address>']
)

Step 5: Handle scenarios where EIP is not owned

IP addresses that are found to be not owned by you can be handled in multiple ways. They could be either output to an S3 bucket, or be notified to a group using an SNS event. Alternatively, as a remediation process you could invoke the change_resource_record_sets() method to delete the violating record, for which you can use the aws-controltower-AuditAdministratorRole role. In our case, an Audit report of the findings is output into an S3 bucket; each record in the output contains the AWS account#, the Hosted Zone ID, the Record Set Name, the IP Address, and if the IP is owned by the account or not.

Other sample use cases

We described the dangling DNS records use case in detail as part of this post. Here are some other examples of potential use cases that you can address using this approach –

Determine which AWS account owns a particular public IP address or an Elastic IP address.
Find out how many AWS accounts have Security Hub, GuardDuty or similar services turned on.
Enable VPC Flow logs across all your accounts.
Gather an inventory of gp2 volumes across all your accounts that could potentially be migrated to gp3.

While the examples above may suggest that you want to always read or update all accounts in your AWS Organizations, you can also target a smaller subset of accounts for very specific use cases that may not apply broadly across all your accounts. You can potentially discover several other use cases where this mechanism can be employed.

Conclusion

AWS Management and Governance services provide solutions to many common use cases and we are constantly enhancing them to support your governance, security, and compliance needs with minimal effort. In this blog, we described how you can use AWS Control Tower Audit account to inspect or affect changes across different accounts in your AWS Organizations, using AWS Lambda. We demonstrated this with a sample implementation for a use case. This mechanism provides you a convenient option to systematically manage your multi-account use cases. You can take advantage of the roles and relationships that Control Tower creates for you, so you don’t need to develop a custom solution for this purpose.

Additional Resources

About the authors:

AWS Cloud Operations Blog