In an earlier blog, Taming client-side key rotation with the Amazon S3 encryption client, we introduced the putInstructionFile API that makes Amazon S3 client-side key rotation easy. In the long run, however, wouldn’t it be nice if you could eliminate the administrative overhead of managing your client-side master keys, and instead have them fully managed and protected by a trusted, secure, and highly available key management service?

This is exactly where the recently launched AWS Key Management Service (KMS) can help. In this blog, we will provide an example of how you can leverage the putInstructionFile API to migrate from the use of an S3 client-side master key to the use of a KMS-managed customer master key (CMK).  In particular, this means you can re-encrypt your existing S3 data keys (aka envelope keys) with a different master key without touching the encrypted data, and ultimately retire and remove the need to manage your own client-side master keys.

Let’s look at some specific code.

Pre-Key Migration to AWS KMS

Suppose you have a pre-existing Amazon S3 client-side master key used for Amazon S3 client-side encryption. The code involved to encrypt and decrypt would typically look like this:

        // Encryption with a client-side master key
        SecretKey clientSideMasterKey = ...;
        SimpleMaterialProvider clientSideMaterialProvider = 
            new SimpleMaterialProvider().withLatest(
                    new EncryptionMaterials(clientSideMasterKey));
        AmazonS3EncryptionClient s3Old = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                clientSideMaterialProvider)
            .withRegion(Region.getRegion(Regions.US_EAST_1));
        
        // Encrypts and saves the data under the name "sensitive_data.txt" to
        // S3. Under the hood, the one-time randomly generated data key is 
        // encrypted by the client-side master key.
        byte[] plaintext = "Demo S3 Client-side Key Migration to AWS KMS!"
                .getBytes(Charset.forName("UTF-8"));
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentLength(plaintext.length);
        String bucket = ...;
        PutObjectResult putResult = s3Old.putObject(bucket, "sensitive_data.txt",
                new ByteArrayInputStream(plaintext), metadata);
        System.out.println(putResult);

        // Retrieves and decrypts the S3 object
        S3Object s3object = s3Old.getObject(bucket, "sensitive_data.txt");
        System.out.println(IOUtils.toString(s3object.getObjectContent()));

In this example, the encrypted one-time data key is stored in the metadata of the S3 object, and the metadata of an S3 object is immutable.

Migrating to AWS KMS

To re-encrypt such a data key using a KMS-managed CMK, you can do so via the putInstructionFile API, like so:

        // Configure to use a migrating material provider that uses your
        // KMS-managed CMK for encrypting all new S3 objects, and provide
        // access to your old client-side master key
        SimpleMaterialProvider migratingMaterialProvider = 
            new SimpleMaterialProvider().withLatest(
                new KMSEncryptionMaterials(customerMasterKeyId))
                .addMaterial(new EncryptionMaterials(clientSideMasterKey));

        AmazonS3EncryptionClient s3Migrate = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                migratingMaterialProvider, config)
            .withRegion(Region.getRegion(Regions.US_EAST_1));

        // Re-encrypt the existing data-key from your client-side master key
        // to your KMS-managed CMK
        PutObjectResult result = s3Migrate.putInstructionFile(
            new PutInstructionFileRequest(
                new S3ObjectId(bucket, "sensitive_data.txt"),
                new KMSEncryptionMaterials(customerMasterKeyId), 
                InstructionFileId.DEFAULT_INSTRUCTION_FILE_SUFFIX));
        System.out.println(result);
        // Data key re-encrypted with your KMS-managed CMK!

Post-Key Migration to AWS KMS

Once the data-key re-encryption is complete for all existing S3 objects (created with the client-side master key), you can then begin to exclusively use the KMS-managed CMK without the client-side master key:

        // Data-key re-encryption is complete. No more client-side master key.
        SimpleMaterialProvider kmsMaterialProvider = 
                new SimpleMaterialProvider().withLatest(
                    new KMSEncryptionMaterials(customerMasterKeyId));
        AmazonS3EncryptionClient s3KmsOnly = new AmazonS3EncryptionClient(
                new ProfileCredentialsProvider(),
                kmsMaterialProvider, config)
            .withRegion(Region.getRegion(Regions.US_EAST_1));

        // Retrieves and decrypts the S3 object
        EncryptedGetObjectRequest getReq = 
            new EncryptedGetObjectRequest(bucket, "sensitive_data.txt")
                .withInstructionFileSuffix(
                    InstructionFileId.DEFAULT_INSTRUCTION_FILE_SUFFIX);
        s3object = s3KmsOnly.getObject(getReq);
        System.out.println(IOUtils.toString(s3object.getObjectContent()));

Why is there the need to use an EncryptedGetObjectRequest in the above example? This is necessary in order to make use of the newly encrypted data key (via KMS) in the instruction file, and not the one (encrypted using your old client-side master key) in the metadata. Of course, had you configured the use of CryptoStorageMode.InstructionFile in the first place, such explicit override during retrieval would not be necessary. You can find such an example (of using the instruction file storage mode) in the earlier blog Taming client-side key rotation with the Amazon S3 encryption client.

That’s all for now. Hope you find this useful. For more S3 encryption options of using AWS KMS, see Amazon S3 Encryption with AWS Key Management Service.