Extend your approved golden AMI with AWS Marketplace Private Image Build

In this post, I will show you how to automate the installation of software from AWS Marketplace onto your golden AMI using Private Image Build, a new feature of AWS Marketplace currently in public beta.

A “golden AMI” or “gold image” is an AMI you standardize through configuration, consistent security patching, and hardening. It also contains agents you approve for logging, security, performance monitoring, etc. Many enterprise customers have a mature AMI pipeline setup to create a golden AMI of base operating systems for the organization. For a sample golden AMI pipeline, see Announcing the Golden AMI Pipeline.

In the past, organizations had to create a golden AMI specific to a software in AWS Marketplace outside of their usual operating system-specific golden AMI workflow because they were starting with a custom AMI from a third-party. They would start with a paid AMI and then harden and customize it. With the public beta availability of Private Image Build, you can now have the option of deploying software products procured from AWS Marketplace onto an existing golden AMI.

What is Private Image Build?

Private Image Build enables you to purchase installable software products through AWS Marketplace and then install those products on a gold image or AMI that you choose from the images available to your AWS account. Private Image Build helps lines of business in an organization ensure that they are complying with organizational best practices by using a company standard golden AMI.

How does Private Image Build work?

Private Image Build uses AWS Marketplace Image Build Service, which runs an AWS Systems Manager Automation in your AWS account. The AWS Marketplace Image Build Service launches an instance of your golden AMI, attaches an InstanceProfile to the instance, then downloads and installs the software package. Afterwards, AWS Marketplace Image Build Service then saves the updated instance as a new image in the customer’s account. Once the installation is complete, AWS Marketplace Image Build Service publishes a status notification on an Amazon Simple Notification Service (SNS) topic you specify. However, if the image creation fails, it lets you see the reason (e.g., incompatible golden AMI operating system, etc.) via the Systems Manager console.

Note

You can use operating system images like Amazon Linux that do not require a subscription. However, the golden AMIs cannot be based on operating system products in AWS Marketplace that require a subscription, such as Centos or Ubuntu.

Here is the architecture diagram for the Private Image Build workflow:

I will now show you how to set up your environment for Private Image Build and also how to deploy software products from AWS Marketplace to your golden AMI.

Setup environment for running Private Image Build

The AWS Marketplace Image Build Service runs in your AWS account and enables installation of your software products on top of your golden AMI. This gives you the flexibility to ask AWS Marketplace Image Build Service to instantiate the base golden AMI in the subnet you specify.

To run successfully, AWS Marketplace Image Build Service requires you to:

1. Create an InstanceProfile that the AWS Marketplace Image Build Service will attach to the instance of your golden AMI.
2. Create an AutomationServiceRole that will run the Systems Manager Automation.
3. Create an SNS topic (optional) to which the service will publish a status notification once the private image has been created.
4. Create an S3 bucket for logs. The S3 bucket name must have image-build as part of it.
5. Use an Identity and Access Management (IAM) user/role that has arn:aws:iam::aws:policy/AWSMarketplaceImageBuildFullAccess policy associated with it. The above policy has access to only those S3 buckets that have image-build present in their name.

To make it easy for you to create these resources, I have created an AWS CloudFormation template you can use. To set up the environment, download the CloudFormation template to your computer. Then, to create a stack:

1. Sign in to the AWS Management Console and choose CloudFormation in the Services menu.
2. Choose Create Stack.
3. On the Select Template page, choose Upload a template to Amazon S3.
4. Choose Choose File and then choose the CloudFormation template you downloaded earlier. Choose Next.
5. On the Specify Details page, specify the Stack Name as Private-Image-Build-Env-Setup.
6. Specify EmailID as your email ID. CloudFormation Stack will create an SNS subscription for this email-id. Once the stack executes, open your mailbox and accept the subscription to receive notifications via email.
7. Specify BucketName. Ensure that your bucket’s name has image-build in it.
8. Choose Next.
9. On the Options page, choose Next.
10. On the Review page, choose the check-box next to the following message: “I acknowledge that AWS CloudFormation might create IAM resources.”
11. Choose Create. The CloudFormation template creates an SNS topic, IAM roles, an Amazon S3 bucket, and an Automation document.
12. On the stacks page, choose Private-Image-Build-Env-Setup.
13. In the Detail pane, choose Outputs to view the output of your stack.

After AWS CloudFormation successfully creates a stack, the Outputs tab displays the following results:

• PIBBucket – The Value box displays the name of the bucket you can specify for storing logs generated by the AWS Marketplace Image Build Service.
• PIBNotificationTopic – The Value box displays an SNS topic’s Amazon Resource Name (ARN) on which the service can publish notifications.
• SampleCommandTemplateForInvokingPIB – The value box displays a sample AWS Command Line Interface (CLI) command template.
• AutomationServiceRole  – The value box displays ARN of AutomationServiceRole the service can use.
• ManagedInstanceRole – The value box displays ARN of ManagedInstanceRole the service can use.

Apart from the artifacts mentioned above, the AWS CloudFormation template also creates a Systems Manager Automation document for your convenience. You can use the GoldenPIBAMIAutomationDoc Automation document if your pipeline is serverless or you need an out-of-the-box solution for scheduling your Private Image Build workflow. The name of the Systems Manager document is available in the Outputs tab of the stack.

• GoldenPIBAMIAutomationDoc– The Value box displays name of the Systems Manager Automation document you can execute to deploy AWS Marketplace software on top of your golden AMI using AWS Marketplace Image Build Service (which is what you will do next).

Next, ensure that you have everything you need for installing the software products from AWS Marketplace onto your golden AMI using Private Image Build.

Prerequisites for installing software using Private Image Build

You need a compatible golden AMI, your software subscription, a fulfillment ID, and your golden AMI’s compatible instance-types to create a new golden AMI from your existing one.

First, you need a golden AMI that is compatible with the software you are trying to deploy. For example, as shown in following image, the listing for this software product indicates that it supports Private Amazon Machine Image (Beta) as a fulfillment method and is compatible with Red Hat Enterprise Linux 7.4.

If you want to extend your golden AMI with the above software, then your golden AMI:

1. Must not be from AWS Marketplace
2. Must be based on Red Hat Enterprise Linux version 7.4.

Next, your AWS account must be subscribed to the software you want to install on your golden AMI. The AWS Marketplace subscription process requires accepting the End User License Agreement (EULA) of the software product on behalf of your organization. Since this has legal implications, I advise you to create a central team in your organization that controls software procurement. The software procurement team should get a legal opinion before making the purchase. For more information on access and best practices around marketplace subscriptions, see Securing access to AMIs in AWS Marketplace.

Next, you need to know the Fulfillment Option ID. You can find this information on the Configure this software page of the listing as highlighted in the image below.

Deploying software onto your golden AMI using Private Image Build

Now, you are ready to deploy your AWS Marketplace software on top of your golden AMI using Private Image Build. You have following three options:

Option 1: Trigger installation of software on your golden AMI via AWS CLI

Private Image Build is available in public beta and as of now, you need to download the model and configure it with your AWS CLI.

First, download the marketplaceimagebuild-2017-12-15.normal-1.json file on your build server. Next, execute the following command to register it with your AWS CLI after replacing Enter-complete-path-here with the appropriate local file path.

aws configure add-model --service-model file://<Enter-complete-path-here>/marketplaceimagebuild-2017-12-15.normal-1.json --service-name marketplaceimagebuild

Now that your AWS CLI is ready, you can use the value of SampleCommandTemplateForInvokingPIB, you noted earlier, as a starting point for triggering the AWS Marketplace Image Build Service.

Here is a sample output printed in output section for SampleCommandTemplateForInvokingPIB
that you noted earlier. Ensure that you replace values marked in blue with the appropriate values before you the run the command on your build server.

aws marketplaceimagebuild start-build

--input-fulfillment-option-ids "specify-fulfillment-ids-here"

--output-image-name "specify-target-image-name-here"

--input-image-id "specify-image-id-here"

--output-image-description "specify-image-description-here"

--input-instance-type "specify-instance-type-here"

--output-installation-log-s3-bucket-name "XXX-XXXXXXXXXXX"

--input-automation-role "XXX-AutomationServiceRole-XXXXXXXXXXX"

--input-instance-profile "XXX-ManagedInstanceProfile-XXXXXXXXXXX "

--region "specify-region-here"

--input-sns-topic-arn "arn:aws:sns:region:XXXXXXXXXXX:XXXXXXXXXXX"

Notes:

1. The start-build CLI command shown does not specify a subnet-id. However, the command has an option to specify a subnet-id if you would like your Private Image Build to instantiate your golden AMI in your subnet. To know what all options are available as part of aws marketplaceimagebuild start-build CLI command, add marketplaceimagebuild model to your CLI and type following command

aws marketplaceimagebuild start-build help

2. Apart from CLI, marketplaceimagebuild is also available as part of Java SDK.

Option 2: Trigger the process using an Automation document

If you do not have a dedicated build server and you use a golden AMI pipeline like the one available here, you can customize and reuse some of the steps from the sample Automation document specified in the value section of GoldenPIBAMIAutomationDoc you noted earlier. You can schedule the Automation document to run at a specified interval or trigger it as part of your CI/CD pipeline. To know more about how to set up a CI/CD pipeline, see how to set up a CI-CD pipeline on AWS.

Here is the workflow diagram for triggering the process via a custom Automation document:

The Automation document starts a server and then follows instructions specified in the Trigger installation of software on your golden AMI via AWS CLI section of this blog post to initiate the Private Image Build workflow. Before you run the Automation document, review CLI coded as part of the Automation document. To learn how to trigger an Automation document, see documentation to run a sample Automation workflow.

Option 3: Trigger the process in AWS Marketplace.

For instructions, please see the Building a private image section in the Private Image Build customer guidance document.

Viewing, securing, and distributing your golden AMI

Once your private image has been created, AWS publishes a notification on the SNS topic you specified. You can view the private paid golden AMI by opening https://thinkwithwp.com/ec2/ (EC2) service console and navigating to AMIs. The AMI will appear under Owned by me section of the console. Next, you need to ensure that your paid golden AMI is secure. To know more about how to secure a paid AMI, see Securing access to AMIs in AWS Marketplace.

Distribution of your golden AMI that includes software products from AWS Marketplace can also happen via AWS Service Catalog. To learn more about how to distribute an AMI through AWS Service Catalog, see Use AWS Service Catalog to build a custom catalog of products from AWS Marketplace.

There is a software and infrastructure charge for the AWS services that you use to complete the build process, which may take one or two hours depending on the product. However, there is no additional charge for using the AWS Marketplace Image Build Service to create private images. Golden AMIs built using Private Image Build are only shared with your organization, and software charges are reflected on your AWS bill, just as with pre-built AMIs from AWS Marketplace. Once the image is built, you don’t incur charges for product or AWS resource usage until you use the software product.

Conclusion

In this blog post, I have shown you how to automate installation of a software product from AWS Marketplace directly onto your golden AMI using Private Image Build.

There are multiple software products in AWS Marketplace that are available (here) via Private Image Build today, and the list is growing. To find out more about how to use Private Image Build with products in AWS Marketplace, see the Private Image Build customer guidance document. The AWS Marketplace team is also looking for feedback on products you would like to see enabled for Private Image Build. To send a recommendation, reach out to: awsmp-private-image@amazon.com.

If you have questions about implementing the solution described in this post, please contact AWS Support.

About the author

Kanchan Waikar is an AWS Marketplace Solutions Architect at Amazon Web Services. She enjoys helping customers build architectures through AWS, AWS Marketplace, and AWS Service Catalog.