Disabling ACLs for existing Amazon S3 workloads with information in S3 server access logs and AWS CloudTrail

Access control lists (ACLs) are permission sets that define user access, and the operations users can take on specific resources. Amazon S3 was launched in 2006 with ACLs as its first authorization mechanism. Since 2011, Amazon S3 has also supported AWS Identity and Access Management (IAM) policies for managing access to S3 buckets, and recommends using policies instead of ACLs. While ACLs continue to be fully supported in S3, most use cases no longer require them.

In 2021 we introduced S3 Object Ownership, which allows customers to fully disable ACLs for their S3 buckets and to rely entirely on policies for access control. Disabling ACLs is an S3 security best practice, and in April of 2023, we disabled ACLs by default for all new buckets. However, for existing workloads customers tell us that they need visibility into how their ACLs are being used, before switching over to policies. They want to be sure that this change won’t remove needed permissions, and therefore disrupt their applications.

To help you disable ACLs, we’ve added information to S3 server access logs and AWS CloudTrail so that you have visibility into requests that were dependent on an ACL for access to objects in S3. In this blog post, we will show how you can use this information to disable ACLs with confidence. Following these steps, you will be able to take advantage of the increased scalability and flexibility that policies offer for managing access to resources. Consolidating object-specific permissions into one policy is easier to audit and update than multiple ACLs, particularly at scale.

Solution walkthrough

Here is a high-level overview of what we will cover in this blog post:

1. Enable logging for your bucket
2. Use Amazon Athena to query the logs and identify S3 requests that depend on ACLs
3. Migrate ACL permissions to bucket policy permissions
4. Validate your readiness to disable ACLs
5. Disable ACLs

We will be demonstrating Steps 1 and 2 with AWS CloudTrail, but you can also enable and query S3 Server Access Logs and then continue with Steps 3-5 in this blog post.

Step 1: Enable logging for your bucket

To begin, you must enable AWS CloudTrail data events for your bucket and create an Amazon Athena table. If you have already enabled CloudTrail for your bucket and created an Athena table, you can proceed to Step 2: Use Amazon Athena to query logs. Otherwise, complete the following steps:

1. Enable AWS CloudTrail data events
2. Create an Amazon Athena Table

You will want to wait some period of time that is dependent on your workload before proceeding with the following steps, so that data events have been logged for the S3 bucket you configured for CloudTrail. For example, if you have daily and weekly workloads accessing data in your bucket, you may want to wait at least a week to collect all access patterns.

If you are using S3 server access logs, please see the documentation on enabling S3 server access logging.

Step 2: Use Amazon Athena to query logs and identify S3 requests that depend on ACLs

In this step, we’ll cover how ACL-dependent requests are logged in AWS Cloudtrail and how you can query logs with Amazon Athena.

How ACL-dependent requests are logged in AWS CloudTrail

The new aclRequired field in Amazon S3 server access logs and AWS CloudTrail gives you information on each S3 request to indicate whether or not the request required an ACL for authorization. Its value is either “Yes” or absent in AWS CloudTrail. The purpose of this field is to show you which requests will require a modification to your bucket policy or requests before you can disable ACLs. Here are some examples:

Example 1: A cross-account request by account 222 to an object owned by 111, where an ACL allowing access to 222 is present.

If there is no bucket policy statement allowing access to account 222, the request depends on an ACL in order to succeed, and aclRequired is "Yes". The following is a sample AWS CloudTrail log for this example.

"userIdentity": {
    "accountId": "222",
},
"eventTime": "2022-11-17T19:46:01Z",
"eventName": "GetObject",
"resources": [
  {
    "type": "AWS::S3::Object",
    "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/example-object"
  },
  {
    "accountId": "111",
    "type": "AWS::S3::Bucket",
    "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
  }
],
...
"additionalEventData": {
  "aclRequired": "Yes"
}

If the bucket policy does allow access to account 222, removing the ACL would not interrupt access, and aclRequired is absent.

Example 2: Same-account requests (the object owner and the caller are in the same AWS account)

For same-account read operations (e.g. Get, List), ACLs are not used for authorization, so aclRequired is absent. The following is a sample AWS CloudTrail log for this example.

"userIdentity": {
    "accountId": "111",
},
"eventTime": "2022-11-17T19:46:01Z",
"eventName": "GetObject",
"resources": [
    {
        "type": "AWS::S3::Object",
        "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/example-object"
    },
    {
        "accountId": "111",
        "type": "AWS::S3::Bucket",
        "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
    }
],
...

Example 3: Same-account requests that set an ACL on an object

For same-account PUTs that do not set an ACL or do set a Bucket Owner Full Control ACL, aclRequired is absent.

All other PUT Requests that set an ACL on an object, such as PutObjectACL, will show aclRequired is “Yes”. The following is a sample AWS CloudTrail log for this example.

"userIdentity": {
    "accountId": "222",
},
"eventTime": "2022-11-17T19:46:01Z",
"eventName": "PutObjectACL",
"requestParameters": {
  "ObjectCannedACL": "BucketOwnerFullControl",
},
"resources": [
  {
    "type": "AWS::S3::Object",
    "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/example-object"
  },
  {
    "accountId": "222",
    "type": "AWS::S3::Bucket",
    "ARN": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
  }
],
...
"additionalEventData": {
  "aclRequired": "Yes"
}

The requests marked with a “Yes” in this field indicate where you need to take additional actions before you can disable ACLs. Next, we will show how to query the logs to identify these requests. For a more detailed look into mappings between operations and the expected aclRequired value in CloudTrail or server access logs, see this overview of access control lists (ACLs).

Querying CloudTrail logs with Athena

Once you have waited for data events to be logged in CloudTrail, you can run the query below to find all the ACL-dependent S3 requests over a time period of your choice. If you access data frequently, we recommend first testing out this query by selecting a shorter time period, such as a few hours, to get a sampling. In the following example, we use a 3-day period, but you should modify the query to fit a timeframe that captures all of your application’s workloads to your bucket at least once. This query generates relevant information, such as the names of the events, number of events, the requestor account IDs, and the bucket names. It only retrieves information from the time you enabled logging on the bucket and after the aclRequired field was launched in all regions (2/15/23).

SELECT 
   eventName,
   COUNT(*) AS eventCount,
   userIdentity.accountId, 
   json_extract_scalar(requestParameters, '$.bucketName') as bucketName 
 FROM cloudtrail_table
 WHERE 
   json_extract_scalar(additionalEventData, '$.aclRequired') = 'Yes' 
   AND eventTime > to_iso8601(current_date - interval '3' day)
   AND errorCode IS NULL
GROUP BY 
   userIdentity.accountId, 
   eventName, 
   json_extract_scalar(requestParameters, '$.bucketName') 
ORDER BY userIdentity.accountId;

If you want to look at S3 requests over a specific time window, here is an example query you can use.

SELECT 
   eventName,
   COUNT(*) AS eventCount,
   userIdentity.accountId, 
   json_extract_scalar(requestParameters, '$.bucketName') as bucketName 
 FROM cloudtrail_table 
 WHERE 
   json_extract_scalar(additionalEventData, '$.aclRequired') = 'Yes' 
   AND eventTime > '2023-02-15T00:00:00Z' AND eventTime < '2023-02-18T00:00:00Z'
   AND errorCode IS NULL
GROUP BY 
   userIdentity.accountId, 
   eventname, 
   json_extract_scalar(requestParameters, '$.bucketName') 
ORDER BY userIdentity.accountId;

Here is an example output of this query:

The output of this query indicates that PutObject, GetObject, and ListBucket requests from account 111122223333 depend on ACLs to succeed. To be able to follow the security best practice of disabling ACLs, we need to allow access to these users in the bucket policy and verify ACLs are no longer being set on requests, which we will do in Step 3: Migrate ACL permissions to bucket policies and Step 4: Validate your readiness to disable ACLs.

You might come up with an empty result from this query. This means that during this time period, no requests would have been rejected with ACLs disabled. If your query is empty, you can verify this is true for a different time period as well, or proceed to Step 5: Disable ACLs.

Querying S3 server access logs with Athena

If you have enabled server access logs for your bucket, see Using Amazon S3 access logs to identify requests for the equivalent Athena query.

Step 3: Migrate ACL permissions to bucket policies

Using the output from the Athena queries, we can now write a bucket policy statement that will make these ACLs no longer required for authorization.

In Step 2, we identified three operations PutObject, GetObject, and ListBucket, made by account 111122223333 against the bucket “DOC-EXAMPLE-BUCKET”. These operations must be translated to actions in the bucket policy’s Action element and specify the appropriate Resource as shown in the following bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "111122223333"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
            ]
        }
    ]
}

To write this policy, we mapped each of the operations in our query results to the respective IAM action and resource. The following is a chart with some common S3 operations and their equivalent bucket policy actions and resource types. For a more detailed look into mappings between operations and bucket policy actions and resources, see the documentation on actions, resources, and condition keys for AWS Services.

Note: Operations with n/a bucket policy action must stop setting ACLs. See Step 4.

With this bucket policy, we granted account 111122223333 access to all of the objects within the bucket. While that covers our known access patterns, if you know account 111122223333 actually needs access only to the data under sample-prefix/*, you can write a more specific policy statement giving access to only the specific prefix. See Bucket policy examples for examples of refinements on the above bucket policy statement.

Step 4: Validate your readiness to disable ACLs

After ACLs are disabled, all PutBucketACL and PutObjectACL, along with APIs which set ACLs that are not the Bucket Owner Full Control ACL, will fail with 400 errors. To address this, all PutBucketACL and PutObjectACL requests must be removed and all other APIs which specify an ACL should remove the ACL argument from the request. Below are examples of these changes.

// Removing custom ACL from PutObject Request
final PutObjectRequest putObjectRequest = PutObjectRequest.builder()
        .bucket(bucketName)
        .key(objectName)
        .grantRead(aclString)
        .build();

// Removing canned ACL from PutObject Request
final PutObjectRequest putObjectRequest = PutObjectRequest.builder()
        .bucket(bucketName)
        .key(objectName)
        .acl(aclString)
        .build();

Validating that ACLs are no longer required

Once the necessary changes have been made to your bucket policy, and where needed, client applications, you should wait some time to allow subsequent requests to be logged. Just as in Step 1: Enable logging for your bucket, the amount of time that is appropriate to wait for this analysis might be days or weeks, depending on your workload. Next, you can re-evaluate your logs using a query such as the one below that queries logs since you made the update to be sure that you now have no workloads depending on ACLs. If the query returns zero results starting from the date the bucket policy was applied, you can disable ACLs without disrupting existing workloads.

SELECT 
   eventName,
   COUNT(*) AS eventCount,
   userIdentity.accountId, 
   json_extract_scalar(requestParameters, '$.bucketName') as bucketName 
 FROM cloudtrail_table
 WHERE 
   json_extract_scalar(additionalEventData, '$.aclRequired') = 'Yes' 
   AND eventTime > '2023-02-15T00:00:00Z'
GROUP BY 
   userIdentity.accountId, 
   eventName, 
   json_extract_scalar(requestParameters, '$.bucketName') 
ORDER BY userIdentity.accountId;

The PutObject, GetObject, and ListObjects operations no longer appear in the query results, validating that the permissions have been allowed through the bucket policy and requests do not set ACLs. If you see any results, there are additional ACL-dependent requests unaccounted for by the bucket policy that must be added or ACLs are still being set. If this is the case, repeat Step 3: Migrate ACL permissions to bucket policies and Step 4: Validate your readiness to disable ACLs.

Step 5: Disable ACLs

Now that ACL-dependent permissions have been added to your bucket policy and there are no longer requests in your logs indicating ACLs were required, you can disable ACLs on your S3 bucket with confidence that you won’t disrupt existing workloads.

To disable ACLs for your bucket and the objects in it, you first need to remove any existing bucket ACLs that are in place. You can do this by going to the details page for your bucket in the S3 Console. In the Permissions tab, edit Access control list (ACL) to remove bucket ACLs for any grantees except yourself (the bucket owner). Do not alter the object ACLs.

Once all bucket ACLs are removed, remaining in the Permissions tab, you can disable ACLs on the bucket by editing Object Ownership and setting it to ACLs disabled. This step is reversible, so if you need to re-enable ACLs for any reason, the object ACLs you previously had in place will be immediately restored.

Now you have successfully migrated permissions for your ACL-dependent requests to bucket policy and disabled ACLs on your bucket! Going forward, access to your data is based on policies, including S3 bucket policy and IAM policies.

Cleaning up

To wrap up, complete the following optional steps as part of clean up:

1. Disable AWS CloudTrail data events and/or Amazon S3 server access logs. This is recommended if you do not need logging on the bucket.
2. Delete the corresponding Amazon Athena tables unless you’d like to retain it for future reference.

Most customers can stop here. If for any reason you do not want to disable ACLs, such as if you were simply running tests, you can complete the following steps to re-enable ACLs:

1. Revert the changes made in your bucket’s Permissions tab by editing Object Ownership to ACLs enabled and reverting the bucket policy deletions.
2. Revert the deletion of APIs that set ACLs.
3. Revert the changes made to the bucket policy.

Conclusion

In this blog, we demonstrated how you can use the new aclRequired field in Amazon S3 server access logs and AWS CloudTrail to identify existing applications or access patterns that relied on ACLs for access to your data. The approach shown in this post will allow you to confidently migrate those ACL-dependent permissions to equivalent bucket policies, allowing you to disable ACLs.

Another best practice we recommend is to enable Block Public Access. This feature allows you to limit public access to your S3 bucket. For more on BPA, see Blocking Public Access to your Amazon S3 Storage.

Thanks for reading this blog post. If you have any comments or questions, don’t hesitate to post a comment in the comments section.

Additional resources

• Heads-Up: Amazon S3 Security Changes Are Coming in April of 2023 (S3 Block Public Access enabled by default and ACLs disabled by default for new buckets)
• New – Simplify Access Management for Data Stored in Amazon S3 (S3 Object Ownership to disable ACLs)