Taming client-side key rotation with the Amazon S3 encryption client

As mentioned in an earlier blog, encrypting data using the Amazon S3 encryption client is one way you can provide an additional layer of protection for sensitive information you store in Amazon S3. Under the hood, the Amazon S3 encryption client randomly generates a one-time data encryption key per S3 object, encrypts the key using your client-side master key, and stores the encrypted data key as metadata in S3 alongside the encrypted data. In particular, one interesting property of such client-side encryption is that the client-side master key is always present only locally on the client side, is never sent to AWS, and therefore enables a high level of security control by our customers.

Every now and then, however, an interesting question arises: How can a user of the Amazon S3 encryption client perform key rotation on the client-side master key? Indeed, for security-conscious customers, rotating a client-side master key from one version to the next can sometimes be a desirable feature, if not a strict security requirement. On the other hand, due to the immutability of the S3 metadata, which is where the encrypted data key is stored by default, it may seem necessary to copy the entire S3 object just to allow re-encryption of the data key. And for large S3 objects, that seems rather inefficient and expensive!

In this blog, we will introduce an existing feature of the Amazon S3 encryption client that makes client-side master key rotation feasible in practice. The feature is related to the use of CryptoStorageMode.InstructionFile. In a nutshell, if you explicitly select InstructionFile as the mode of storage for the meta information of an encrypted S3 object, you would then be able to perform key rotation via the instruction file efficiently without ever touching the encrypted S3 object.

The key idea is that, by using an instruction file, you can perform efficient key rotation from one client-side master key to a different client-side master key. The only requirement is that each of the client-side master keys must have a 1:1 mapping with a unique set of identifying information. In the Amazon S3 encryption client, this unique set of identifying information for the client-side master key is called the “material description.”

A code sample could be worth a thousand words. :) To begin with, let’s construct an instance of the Amazon S3 encryption client with the following configuration:

An encryption material provider with a v1.0 client-side master key for S3 encryption
CryptoStorageMode.InstructionFile
Not to ignore any missing instruction file of an encrypted S3 object. (More on this below.)

// Configures a material provider for a v1.0 client-side master key
SecretKey v1ClientSideMasterKey = ...;
SimpleMaterialProvider origMaterialProvider = new SimpleMaterialProvider().withLatest(
    new EncryptionMaterials(v1ClientSideMasterKey).addDescription("version", "v1.0"));

// Configures to use InstructionFile storage mode
CryptoConfiguration config = new CryptoConfiguration()
            .withStorageMode(CryptoStorageMode.InstructionFile)
            .withIgnoreMissingInstructionFile(false);

final AmazonS3EncryptionClient s3v1 = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                origMaterialProvider, config)
            .withRegion(Region.getRegion(Regions.US_EAST_1));

Now, we are ready to use this encryption client to encrypt and persist objects to Amazon S3. With the above configuration, instead of persisting the encrypted data key in the metadata of an encrypted S3 object, the encryption client persists the encrypted data key into a separate S3 object called an “instruction file.” Under the hood, the instruction file defaults to use the same name as that of the original S3 object, but with an additional suffix of “.instruction”.

So why do we need to explicitly set the IgnoreMissingInstructionFile to false? This has to do with the eventual consistency model of Amazon S3. In this model, there is a small probability of a momentary delay in the instruction file being made available for reading after it has been persisted to S3. For such edge cases, we’d rather fail fast than to return the raw ciphertext without decryption (which is the default behavior for legacy and backward compatibility reasons). The eventual consistency model of Amazon S3 also means there are some edge cases that you’ll want to watch out for when updating the S3 data object; but we’ll cover that in one of our upcoming posts.

To continue,

// Encrypts and saves the data under the name "sensitive_data.txt"
// to S3. Under the hood, the v1.0 client-side master key is used
// to encrypt the randomly generated data key which gets automatically
// saved in a separate "instruction file".
byte[] plaintext = "Hello S3 Client-side Master Key Rotation!".getBytes(Charset.forName("UTF-8"));
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(plaintext.length);
String bucket = ...;
PutObjectResult putResult = s3v1.putObject(bucket, "sensitive_data.txt", new ByteArrayInputStream(plaintext), metadata);
System.out.println(putResult);

// Retrieves and decrypts the data.
S3Object s3object = s3v1.getObject(bucket, "sensitive_data.txt");
System.out.println("Encrypt/Decrypt using the v1.0 client-side master key: "
                + IOUtils.toString(s3object.getObjectContent()));
s3v1.shutdown();

Now, to update the client-side master key from v1.0 to v2.0, we simply specify a different EncryptionMaterials in a PutInstructionFileRequest. In this example, the encryption client would proceed to do the following:

Decrypt the encrypted data-key using the original v1.0 client-side master key.
Re-encrypt the data-key using a v2.0 client-side master key specified by the EncryptionMaterials in the PutInstructionFileRequest.
Re-persist the instruction file that contains the newly re-encrypted data key along with other meta information to S3.

// Time to rotate to v2.0 client-side master key, but we still need access
// to the v1.0 client-side master key until the key rotation is complete.
SecretKey v2ClientSideMasterKey = ...;
SimpleMaterialProvider materialProvider = 
            new SimpleMaterialProvider()
                .withLatest(new EncryptionMaterials(v2ClientSideMasterKey)
                                .addDescription("version", "v2.0"))
                .addMaterial(new EncryptionMaterials(v1ClientSideMasterKey)
                                .addDescription("version", "v1.0"));

final AmazonS3EncryptionClient s3 = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                materialProvider, config)
            .withRegion(Region.getRegion(Regions.US_EAST_1));

// Decrypts the data-key using v1.0 client-side master key
// and re-encrypts the data-key using v2.0 client-side master key,
// overwriting the "instruction file"
PutObjectResult result = s3.putInstructionFile(new PutInstructionFileRequest(
            new S3ObjectId(bucket, "sensitive_data.txt"),
            materialProvider.getEncryptionMaterials(), 
            InstructionFileId.DEFAULT_INSTRUCTION_FILE_SUFFIX));
System.out.println(result);

// Retrieves and decrypts the S3 object using v2.0 client-side master key
s3object = s3.getObject(bucket, "sensitive_data.txt");
System.out.println("Client-side master key rotated from v1.0 to v2.0: "
                + IOUtils.toString(s3object.getObjectContent()));
s3.shutdown();
// Key rotation success!

Once the key rotation is finished, you can now use the v2.0 cilent-side master key exclusively without the v1.0 client-side master key. For example:

// Once the key rotation is complete, you need only the v2.0 client-side
// master key. Note the absence of the v1.0 client-side master key.
SimpleMaterialProvider v2materialProvider =
            new SimpleMaterialProvider()
                .withLatest(new EncryptionMaterials(getTestKeyPair())
                                .addDescription("version", "v2.0"));
final AmazonS3EncryptionClient s3v2 = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                v2materialProvider, config)
            .withRegion(Region.getRegion(Regions.US_EAST_1));

// Retrieves and decrypts the S3 object using v2.0 client-side master key
s3object = s3v2.getObject(bucket, "sensitive_data.txt");
System.out.println("Decrypt using v2.0 client-side master key: "
                + IOUtils.toString(s3object.getObjectContent()));
s3v2.shutdown();

In conclusion, we have demonstrated how you can efficiently rotate your client-side master key for Amazon S3 client-side encryption without the need to modify the existing data keys, or mutate the ciphertext of your existing S3 data objects.

We hope you find this useful. For more information about S3 encryption, see Amazon S3 client-side encryption and Amazon S3 Encryption with AWS Key Management Service.

AWS Developer Tools Blog

Taming client-side key rotation with the Amazon S3 encryption client

Resources

Follow