AWS Storage Blog

Encrypting existing Amazon S3 objects with the AWS CLI

Encryption of data at rest is increasingly required by industry protocols, government regulations, and internal organizational security standards. Encryption helps you protect your stored data against unauthorized access and other security risks.

Amazon S3’s default encryption can be used to automate the encryption of new objects in your bucket, but default encryption does not change the encryption of existing objects in the same bucket. You may have existing objects in your Amazon S3 bucket that must be encrypted, or you may want to change the server-side encryption (SSE) settings you are using. I often get questions from customers on the simplest way to encrypt existing objects in their S3 bucket.

In this post, I cover important things to consider when using the Copy Object API to encrypt existing objects in place. I then provide examples you can use to encrypt existing objects in a bucket to keep your data secure using the AWS Command Line Interface (AWS CLI). I also provide examples you can use to encrypt all S3 objects in a prefix or bucket. Lastly, I discuss common questions around copying and encryption.

Prerequisites

To run the commands outlined in this post, you need:

  • An AWS account
  • At least one Amazon S3 bucket
  • The AWS CLI

Things to know

First thing’s first, BE CAREFUL! To encrypt an existing object using SSE, you replace the object. To encrypt existing objects in place, you can use the Copy Object or Copy Part API. This copies the objects with the same name and encrypts the object data using server-side encryption. Here are some things to consider before using the Copy Object API:

  • LastModified timestamp is changed to the timestamp of the copy. Applications that depend on object timestamps now look at the copy timestamp and not the original upload timestamp. For example, S3 Lifecycle uses the date of the new object now.
  • S3 event notifications can be enabled on PUT or COPY events, and are triggered when you copy an existing object to encrypt it. For example, Lambda functions may be retriggered.
  • S3 Cross-Region Replication (CRR) replicates a new version of the object.
  • Consider your metadata replication needs if you are using one of the following:
    • Access Control List (ACL) is reset to the S3 bucket default. If you use Object ACLs, you must add them in the Copy Operation.
    • Custom metadata: you want to ensure that you are copying the metadata to the new objects. If you are copying objects larger than your multipart_threshold value (5 GB as used below), the AWS CLI does not copy over the metadata. Explicitly set the metadata for the new object with the --metadata parameter.
    • S3 object tags: the AWS CLI cp command does not copy object tags from the source object to the destination object. You can use the aws s3api put-object-tag command to add these to the new object.
  • S3 ETag may change.
    • Multipart uploaded objects: if you use a different multipart chunk size for the copy compared to the original multipart upload, the ETag changes.
    • If you encrypt the object with SSE-KMS (server-side encryption using AWS Key Management Service to encrypt data) or SSE-C (server-side encryption with customer-provided encryption keys), the object will not have the same ETag as before.
  • Bucket versioning: in buckets that have versioning enabled this procedure creates a new version of the object that is encrypted, but does not modify the existing unencrypted object version.
  • The storage class defaults to STANDARD. If you want a different storage class, you can set this with --storage-class. This also makes it an easy way to modify the storage class that your objects are using.
  • Object Lock: If you are using object lock the retention period is reset to the bucket default.

Suggestions before attempting encryption using the AWS CLI

Outside of the above there are several other things that you should consider before attempting encryption using the AWS CLI:

  • Consider enabling S3 Versioning or other ways that you could revert this change.
  • If you are planning to use SSE-KMS, ensure that users or applications that are accessing this data have the correct permissions.
  • Perform this on a subset of your data first before applying it to all the objects in your S3 bucket.
  • Consider setting up the S3 Inventory to monitor what are encrypted, as well as other details that you may want to consider.

Encrypting objects using the AWS CLI

To get started, you must install and configure the AWS CLI. What follows is a collection of commands you can use to encrypt objects using the AWS CLI:

  • You can copy a single object back to itself encrypted with SSE-S3 (server-side encryption with Amazon S3-managed keys) using the following command:
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse AES256
  • You can copy a single object back to itself encrypted with SSE-KMS using the default Amazon S3 key with the following command:
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse aws:kms
  • You can copy a single object back to itself encrypted with SSE-KMS using a customer managed key by adding the --sse-kms-key-id argument:
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse aws:kms --sse-kms-key-id arn:aws:kms:us-west-2:111122223333:key/3aefc301-b7d2-4601-9298-5a854cf9999d
  • You can also see what the command does before running with the --dryrun flag.
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse aws:kms --dryrun  
(dryrun) copy: s3://awsexamplebucket/myfile to s3://awsexamplebucket/myfile
  • If you must encrypt all objects in your S3 bucket, you can run the following command:
aws s3 cp s3://awsexamplebucket/ s3://awsexamplebucket/ --sse aws:kms --recursive
  • You can copy all objects in a prefix with the following command:
aws s3 cp s3://awsexamplebucket/prefix1/ s3://awsexamplebucket/prefix1/ --sse aws:kms --recursive
  • If you have a large number of objects, you can speed up the copy process by increasing the number of threads and/or chunk size that the AWS CLI uses. Please see configuration for a detailed explanation of these:
aws configure set default.s3.max_concurrent_requests 60
aws configure set default.s3.multipart_threshold 5GB
aws configure set default.s3.multipart_chunksize 5GB

More options and examples for copying and configuration can be found in the AWS CLI documentation.

Common questions around copying and encryption

Here I run down a list of common questions that customers have around copying and encryption.

How long does copying take?

I tested from a T2.medium EC2 instance in the same Region as the S3 bucket. I used a total of 10,000 1-GB objects for a total size of 10 TB. I was able to complete encrypting all objects in my test bucket in minutes using the SSE-KMS encryption type.

The amount of time it takes to copy varies, with the variance primarily based on total object counts. For example, a large number of small objects takes longer than a small number of large objects even if the total size is greater.

What about large buckets?

For S3 buckets with a large number of objects, in the order of millions or billions of objects, using Amazon S3 inventory or Amazon S3 Batch Operations can be a better option than using the AWS CLI instructions in this post. Check out this blog post to learn more about batch operations.

What about data stored in Amazon S3 Glacier or Amazon S3 Glacier Deep Archive?

When objects are moved into Amazon S3 Glacier or Amazon S3 Glacier Deep Archive, they are automatically encrypted at rest. Objects in S3 Glacier or S3 Glacier Deep Archive must first be restored before executing the copy request to encrypt them with SSE-S3.

What about versioned objects?

When you overwrite an S3 object, it results in a new object version in the bucket. Running the commands outlined here results in a new encrypted version. However, it does not remove the old unencrypted version of the object. If you want to remove these versions, see the versioning documentation to understand how to use S3 Lifecycle to expire previous versions of objects.

What about tags, ACL, or custom metadata?

If an object is greater than your multipart_threshold (5 GB as used in this example), the AWS CLI is unable to copy the existing Tags, ACL, or Custom metadata from the source object. In this case, we use the 5 GB max, but if your object is larger than you must explicitly add them.

  • As an example, for an S3 object tag this would be:
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse aws:kms --metadata tag1=value1
  • As an example, for an S3 object ACL this would be:
aws s3 cp s3://awsexamplebucket/myfile s3://awsexamplebucket/myfile --sse aws:kms --acl bucket-owner-full-control

More examples and information can be found in the AWS CLI documentation.

What would encryption of existing objects using the CLI cost?

As we are not moving any data outside of S3, there is no additional data transfer costs incurred for the encryption by using the CLI instructions in this post. You only incur the costs of the LIST and COPY API Calls, and if using SSE-KMS, the cost of encrypting objects. If you do not delete the previous version of your now encrypted objects, you will be charged for the storage of both versions of the objects.

Cleaning up

After completing the encryption steps outlined in the post, you want to reset the AWS CLI settings to their defaults or some value that is optimized for your use case.

These commands return your CLI settings to default:

aws configure set default.s3.max_concurrent_requests 10
aws configure set default.s3.multipart_threshold 8MB
aws configure set default.s3.multipart_chunksize 8MB

More CLI configuration information and options can be found here.

Conclusion

In this post, I demonstrated how to use the AWS CLI to encrypt existing data in your Amazon S3 buckets to help ensure that your data is protected. I also covered several things to consider when encrypting your objects, as well as a few suggestions. Running these commands allows you to quickly and easily complete operations like encrypting all existing objects in your S3 bucket to meet compliance, or internal, guidelines. This can save you time setting up your encryption while enabling you to achieve high levels of data security.

Thanks for reading this blog post about encrypting objects in Amazon S3 using the AWS CLI, please leave a comment if you have any feedback or questions!