Unlocking Benefits with Bottlerocket: A Purpose-Built Container OS

This post discusses Bottlerocket, a Linux-based open source operating system (OS) that is purpose-built for running containers. We walk through fleet management at scale and how users can benefit from it. We also share how the product roadmap of Bottlerocket is driven based on community and user feedback.

The challenges of fleet management with General-Purpose host Operating Systems

General-Purpose Operating Systems (GPOS) are designed to accommodate a wide range of applications and use cases, providing a versatile environment for diverse computing needs. However, this inclusivity presents challenges in terms of optimization and fleet management. In environments with a large GPOS footprint, the multitude of software packages often leads to version disparities across systems, making fleet management complex. Then, coordinating updates and making sure of uniformity become formidable task, especially in large-scale deployments where compatibility must be preserved for a wide variety of applications.

To reduce some of the compatibility challenges, GPOS vendors typically offer long-term support (LTS): a product lifecycle management policy not to make incompatible changes to software packages, combined with a promise to provide security and bug fixes. It is difficult to keep the compatibility promise while continuously updating open source software to the latest versions, and expensive to keep the support promise by back-porting patches and fixes to older versions. These challenges and costs grow over time, which leads to typical LTS lifetimes of five to ten years for a GPOS, with the user migrating to a newer version before the support period ends.

Motivation for Single-Purpose Operating Systems

Single-Purpose Operating Systems (SPOS) are designed and optimized for specific tasks. They are common in embedded environments or devices with a constrained user interface or API, such as a wireless router. They may not support remote login or interactive shell access except through specialized debugging mechanisms. They are often image-based rather than package-based. On package-based systems, software is decomposed into packages that can be installed and updated independently by a package manager. On image-based systems, software is pre-installed into a filesystem image and must be installed and updated all at once. This uniformity minimizes the potential delta between the state of the system during pre-release testing, and the actual end state after applying an update on production systems, which is a frequent source of user pain.

For an SPOS, the software it ships is an implementation detail on which users cannot take a direct dependency. It is still bounded by Hyrum’s Law: all observable behaviors of the system are depended upon by someone, but the OS can restrict access to the specific library versions and command-line programs installed, and mandate the use of the application programming interface (API) instead. This means that the majority of software can be updated to the latest versions without breaking compatibility promises to users. This minimizes the expense of ongoing support. Consequently, an SPOS might not define an LTS lifetime at all.

SPOS for container hosts

Containers are a popular solution for the fleet management problem in GPOS environments. Each host in the fleet can run multiple containers, and each container, while sharing the same kernel, can include a different set of packages with different configurations depending on the needs of the application. The trade-off is that it is no longer possible to update a package on the host and have that update affect all the running applications. The benefit is that fleet-wide package updates no longer need to move at the pace of the slowest-moving application. In addition, once all of the applications run inside containers, the packages installed on the host no longer matter, which means the host may not need a package manager at all. This freedom makes it possible to use an SPOS to further streamline fleet management when running container workloads.

Moving to image-based deployments eliminates the problem of diversity at the host level, which in turn streamlines fleet management in two ways:

• by minimizing the potential for untested combinations of software, and
• by eliminating the risk of unexpected local edits.

Furthermore, If the SPOS does not define an LTS and does not need migrations, then fleet management can be further streamlined.

Introduction to Bottlerocket

Bottlerocket is an open source, container-optimized Linux distribution – an SPOS designed for the exclusive purpose of hosting containers. Each Bottlerocket image, known as variant, is specialized for a specific combination of container orchestrator and cloud environment. Bottlerocket variants are provided to users as a set of Amazon Machine Images (AMI), currently optimized for either Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS), with an associated stream of images for in-place updates. These AMIs are available for use at no additional cost, and are covered by AWS support plans. The end-of-life for a given variant is tied to a software lifecycle that falls under shared responsibility of users, such as minor versions of Kubernetes or newer versions of the Linux kernel. Otherwise, at an operating system level, Bottlerocket does not have specific end-of-life policy.

Bottlerocket images do not have an SSH server nor even a shell. Instead, they are API-driven and offers an API to adjust system settings and to trigger actions such as a reboot. Containers are the only supported mechanism for running custom code on Bottlerocket. Bottlerocket facilitates systems management tasks by providing two optional add-on containers: one for AWS Systems Manager (SSM) based management, which is enabled by default and another for Secure Shell (SSH) access, which is disabled by default.

Reasons users choose Bottlerocket

Users experience benefits such as increased uptime for container applications, improved security and resource usage, lower management overhead and operational cost, and faster node ready times. This is reflected in the customer and partner testimonials on the Bottlerocket Getting Started page. We believe that these benefits can be attributed to the core Bottlerocket values of minimalism, operational reliability, and out-of-the-box compliance.

Security benefits from curated dependencies: The Bottlerocket team maintains around 100 package definitions, as compared to 50,000 in a large GPOS. Having full control over each package definition makes sure that unnecessary features and dependencies are disabled at build time. With a few exceptions, such as the kernel, all software can be updated to the latest upstream version at any time, since none of it is directly exposed to users. These attributes make it easier to avoid vulnerabilities in the first place, and to fix the ones that are identified.

Operational reliability through image-based updates: Bottlerocket offers In-Place Updates where both upgrade and rollback are fully tested before reaching users. This is a core product value that’s reflected in the supporting infrastructure: AMIs for every supported version of Bottlerocket can be discovered through long-lived Systems Manager parameters, and any running Bottlerocket instance can be upgraded or downgraded to any version.

Compliance, out-of-the-box:A golden AMI is an AMI that you standardize through configuration, consistent security patching, and hardening. It also contains agents you approve for logging, security, performance monitoring, etc. One of the most common reasons for creating a golden AMI is to harden it according to an industry benchmark, such as those published by the Center for Internet Security (CIS). Bottlerocket offers the only first-party AMIs that are hardened to CIS Bottlerocket Benchmark v1.0.0 optimized for use with Amazon ECS and Amazon EKS, and are fully supported by AWS. It doesn’t support interactive login through SSH or the console, omits any software not necessary for running containers or managing the OS, and enforces cryptographic integrity checks for the root filesystem and mandatory access controls such as SELinux to reduce the attack surface in the event of container escape. This is done even if the compromised application is running as root. Bottlerocket also provides a built-in reporting tool and a Report API so that users can automate the collection of CIS benchmark results.

What if you want to customize the Bottlerocket variant?

The Bottlerocket team builds and maintains a few variants but many Bottlerocket users are interested in creating their own custom Bottlerocket variants. Until recently, additional variants could only be created by forking the Bottlerocket repository and maintaining a series of patches that add the special variant. There are some risks and challenges associated with this method:

• The custom variant repository must frequently re-base a substantial volume of code to stay on top of Bottlerocket changes. These re-bases are unpredictable and likely non-trivial in many cases.
• The integration points of custom variants are not well-defined. For example, changes to the semantic meaning of settings could impact out-of-tree variants in ways that are difficult to discern during re-base.
• The forked repository includes a significant amount of code/documentation/information that is not relevant to the specific changes needed for the custom variant.

We are excited to have recently solved the aforementioned challenges with our Bottlerocket Out-of-Tree Builds (OOTB) support, which is a mechanism for creating custom Bottlerocket variants that live within their own git trees. These variants define a dependency on Bottlerocket’s core in a way that allows the custom variant to evolve independently and consume Bottlerocket core updates safely. We have already migrated existing Bottlerocket variants to OOTB infrastructure. If you built a custom variant of Bottlerocket, then you can now use OOTB to build it without forking and maintaining the entire Bottlerocket core repository. Refer to the design document and we look forward to your feedback through GitHub Issue in this improved approach to build and maintain custom Bottlerocket variants.

Development Plan

The Bottlerocket roadmap is primarily shaped by direct feedback from our users and is based on the following four pillars:

1. Supporting new features across all Amazon Elastic Compute Cloud ( Amazon EC2) instance types and AWS Regions
2. Doubling down on built-in compliance capabilities
3. Making the Bottlerocket ramp-up process easier for users
4. Continuing to improve node ready times

Although the first pillar is self-explanatory, for the second we plan to publish FIPS 140-3 variants in the coming months to make Bottlerocket easier to adopt for customers seeking FedRAMP authorization.

For the third pillar, we are going to streamline the user experience by making boot-time tasks easier to automate. Bottlerocket allows bootstrap containers as a mechanism to perform boot-time tasks such as configuring ephemeral storage. However, today users have to bring their own container image and keep it up-to-date. We plan to create a generic bootstrap container image that executes scripts passed as user data, and can be used by users to configure their instances. This removes the undifferentiated heavy-lifting for users to maintain their own customized solutions for common use cases such as using ephemeral storage to store container images. In addition, we are going to support bootstrap commands that enable users to run predefined commands such as configuring node-local NVM Express (NVMe) storage through the Bottlerocket API without taking runtime dependency on bootstrap containers.

For the final pillar, we are going to continue to improve node launch times. The next minor release includes an optimization contributed by Amazon EKS to ship the “pause” container as part of the OS build. This is a very small container image that must be present on the node before a pod can be created. Although the image pull time for the pause container is minimal, having it already present as a part of OS build makes node ready time even faster. Beyond that, we aim to bring up the network and local storage in parallel, and we plan to switch back to Zstandard (zstd) compressed kernel modules

Call to action

Bottlerocket has a website that serves as a central hub for Bottlerocket related information, tutorials, and other resources to help users get started. If you are new to Bottlerocket, then refer to the Get Started section on the website to learn more about using Bottlerocket as a container host OS.

Bottlerocket is an open source project and we welcome issues with new ideas, bug reports, and pull requests. We’d also love feedback from you about your experiences using Bottlerocket.