Containers

Amazon EKS adds native support for Bottlerocket in Managed Node Groups

Today, Amazon Elastic Kubernetes Services (Amazon EKS) announces native support for Bottlerocket in managed node groups. Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon. It focuses on security and maintainability, and provides a reliable, consistent, and safe platform for container-based workloads. Amazon EKS managed node groups with Bottlerocket support enables you to leverage the simplicity of managed node provisioning and lifecycle management features, while using the latest best practices for running containers in production. You can run your Kubernetes workloads on Bottlerocket nodes and benefit from enhanced security, higher cluster utilization, and less operational overhead.

In this post, we’ll set up an Amazon EKS cluster and launch a Bottlerocket managed node group. We’ll review the managed node group update process and deploy a sample application to show you how to connect to the Bottlerocket node. We’ll also discuss the key considerations to note when working with Bottlerocket managed node groups.

Bottlerocket

Organizations are standardizing on Kubernetes for cloud and on-premises applications. The majority of Amazon EKS customers deploy their applications on cluster nodes. These nodes are powered by general purpose operating systems geared for a range of use cases and include a number of packages that are not required for container orchestration. They may have to update the OS more frequently to address vulnerabilities that develop in these packages over time to meet compliance requirements. Additional software running on the nodes can also result in longer startup times, limiting the ability of clusters to scale computing capacity rapidly in response to application traffic spikes. Furthermore, the extra software consumes node resources such as CPU and memory, resulting in increased compute costs.

To address these issues, AWS launched Bottlerocket, a lightweight, secure, Linux-based open source operating system designed specifically for running containerized workloads. Bottlerocket was built to provide a secure foundation for hosts running containers and minimize operational overhead associated with maintaining nodes at scale. Bottlerocket provides enhanced security via reduced attack surface with enforced permission boundaries. It helps ensure consistency across EKS nodes through image-based updates, a read-only root filesystem, and API-driven configuration. Bottlerocket has mechanisms for performing automatic software updates that are atomic. It also has tools for executing regular management tasks, such as modifying default settings, and tools for emergency scenarios.

Amazon EKS managed node groups

Kubernetes schedules your workloads on nodes, and these nodes need to be provisioned, joined to the cluster, and managed over time for version updates and security patches. Amazon EKS managed node groups is a feature that automates the provisioning and lifecycle management of nodes for Kubernetes clusters. When using managed node groups in Amazon EKS, your Kubernetes nodes are backed by Amazon Elastic Compute Cloud (Amazon EC2) instances in your account, which are managed by an Auto Scaling group. With managed node groups, you don’t need to provision Amazon EC2 instances separately, curate your own Kubernetes node AMIs, or worry about your nodes joining the cluster. You can create, upgrade, and terminate groups of nodes with simple operations in Amazon EKS.

Support for Bottlerocket managed node groups in Amazon EKS enables you to run your applications on container-optimized managed nodes with enhanced security. Bottlerocket is now a built-in managed node group Amazon Machine Image (AMI) option, enabling you to provision container-optimized nodes with a single click. You can leverage existing managed node group notification mechanisms to make updates when newer Amazon EKS Bottlerocket AMIs become available. You can also recycle nodes periodically to standardize management across different node group types.

All of the existing Amazon EKS managed node update behavior now applies to Bottlerocket as well. With managed node group support for Bottlerocket, you can now minimize downtime for your workloads caused by simultaneous node upgrades. You can do this by specifying the maximum number of nodes that can become unavailable during an upgrade in the maxUnavailable field of a node group’s updateConfig. Alternatively, you can select maxUnavailablePercentage, which specifies the maximum number of unavailable nodes as a percentage of the total number of nodes. You can use this to determine the maximum number of instances that will be brought down simultaneously.

Getting started

In this section we will create an EKS cluster, launch Bottlerocket nodes, use SSM agent to connect to control container. We will finish the exercise with updating Bottlerocket nodes.

Prerequisites

  • The AWS CLI with appropriate credentials and session manager installed
  • A default VPC in a Region of your choice. You can also use an existing VPC in your account.
  • A key pair in your account for remote access. The key pair is named eks_bottlerocket.
  • The latest version of eksctl
  • The Amazon EKS-vended kubectl

Create an EKS cluster

Use eksctl to create a cluster. Copy the following configuration and save it to a file called cluster.yaml:

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: bottlerocket-cluster
  region: us-west-2

Now create the cluster:

eksctl create cluster -f cluster.yml

Create a managed node group

You can create Bottlerocket managed node groups for all default (static scaling with T series), general purpose (M series), compute-optimized (C series), memory-optimized (R series) instance types, including Graviton2 ARM-based instance types. Accelerated computing instance types (P, G, Inf1) are not supported. Copy the following configuration to node-group.yaml. Note that SSH is enabled in this configuration for demonstration purposes, and SSM is enabled by default.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: bottlerocket-cluster
  region: us-west-2
  version: 1.20
  
managedNodeGroups:
  - name: bottlerocket-nodegroup
    instanceType: m5.xlarge
    minSize: 2
    maxSize: 4
    desiredCapacity: 3
    amiFamily: Bottlerocket
    labels: { role: br-worker }
    tags:
       nodegroup-type: Bottlerocket 
    ssh:
      allow: true
      publicKeyName: eks_bottlerocket

Now create the node group:

eksctl create nodegroup --cluster bottlerocket-cluster -f node-group.yaml

List the nodes in the EKS cluster along with the attributes of interest to us:

$ kubectl get nodes -o=custom-columns=NODE:.metadata.name,ARCH:.status.nodeInfo.architecture,OS-Image:.status.nodeInfo.osImage,OS:.status.nodeInfo.operatingSystem

NODE                                        ARCH   OS-Image               OS
ip-10-0-101-226.us-west-2.compute.internal  amd64  Bottlerocket OS 1.0.7  linux
ip-10-0-187-156.us-west-2.compute.internal  amd64  Bottlerocket OS 1.0.7  linux

Check the configuration settings of Bottlerocket running on the cluster nodes. Because we already enabled the SSM permission for the node instance role of the nodes, the AWS SSM agent should be running on the node. We can access it through an SSM session and find the instance IDs by running this command:

$ aws ec2 describe-instances --query Reservations[*].Instances[*].[InstanceId] --output text

i-08f2426c826d38ad5
i-068d8f40c6c90a596
i-068d8f40c6c90a798

Start an SSM session

Bottlerocket has a control container enabled by default that runs on a separate instance of containerd. This container runs the AWS SSM agent and enables you to run commands or start interactive shell sessions on Bottlerocket nodes. Choose one of the instances and launch an SSM session:

$ aws ssm start-session --target i-08f2426c826d38ad5

Starting session with SessionId: eks-course-0bf27ea1e078f5c40
Welcome to Bottlerocket's control container!

You are now connected to the Bottlerocket “control” container. To replace this control container with your own, refer to the documentation. In order to access the container through SSM, you need to give the node the appropriate IAM role. Amazon EKS will handle this for you when using a managed node group. Once you have access to the control container, you can execute commands, which in turn make the appropriate API calls to a local service running on the instance to configure and manage your Bottlerocket node. This is not a complete shell environment and you have a limited set of commands available.

The Bottlerocket API includes methods for checking and starting system updates. You can read more about the update APIs in the update system documentation. Most of your interactions will be through the apiclient command. You can find more details in the documentation. The apiclient knows how to handle those update APIs for you. You can also use apiclient to describe the configuration setting on your instance.

$ apiclient -u /settings

We can view key details in its output, such as the node IP address, DNS settings, the motd content, and update URLs. You will also see that the admin container is enabled as we have configured SSH while creating the cluster.

Connect to an instance

Bottlerocket also has an “administrative” container, which is disabled by default and also runs on its own containerd instance on the host. This container has an SSH server running that allows you to log in as ec2-user using your EC2-registered SSH key.

In a terminal window, use the ssh command to connect to the instance. Specify the path and file name of the private key (.pem), and use the public IP from the previous step. To connect to your instance, enter the following command.

ssh -i ~/.ssh/eks_bottlerocket.pem ec2-user@BottlerocketElasticIP

Once inside the admin container, execute sheltie to get a complete root shell into the Bottlerocket instance. You can then invoke other administrative commands. For example, you can collect a set of logs.

sudo sheltie
logdog

This will create a log archive /var/log/support/bottlerocket-logs.tar.gz. You may retrieve the file through SSH. Once you’ve quit the Bottlerocket host, execute the following command:

ssh -i ~/.ssh/eks_bottlerocket.pem \
    ec2-user@BottlerocketElasticIP \
    "cat /.bottlerocket/rootfs/var/log/support/bottlerocket-logs.tar.gz" > bottlerocket-logs.tar.gz

Deploy sample application

Now deploy a sample web application.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    role: webserver
spec:
  replicas: 4
  selector:
    matchLabels:
      role: webserver
  template:
    metadata:
      labels:
        role: webserver
    spec:
      containers:
      - name: frontend
        image: httpd
        ports:
        - containerPort: 80
        command: [ "/bin/sh" ]
        args: ["-c",  "echo '<html> <head> <title>Amazon EKS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center> <h1>Amazon EKS Sample App</h1> <h2>Congratulations!</h2> <p>Your application is now running on a Bottlerocket node in Amazon EKS.</p> </div></body></html>' >  /usr/local/apache2/htdocs/index.html && httpd-foreground"]
EOF

Now, create a service with type LoadBalancer.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  selector:
    role: webserver
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer
EOF

Obtain the URL to our newly deployed service. We may need to wait a couple minutes before the load balancer is successfully provisioned.

kubectl get service sample-app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sampleapp-service LoadBalancer 10.100.14.152 afa7b503ed46f288e96-265681492.us-west-2.elb.amazonaws.com 80:31452/TCP 4m9s

Let’s access the application to see the sample web application in action.

curl http://afa7b503ed46f288e96-265681492.us-west-2.elb.amazonaws.com/ (http://afa7b503ed46f288e96-265681492.us-east-1.elb.amazonaws.com/)
<html> <head> <title>Amazon EKS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center> <h1>Amazon EKS Sample App</h1> <h2>Congratulations!</h2> <p>Your application is now running on a Bottlerocket node in Amazon EKS.</p> </div></body></html>

Upgrade a managed node group

You can use existing update CLI and API to update Bottlerocket node groups. There are different scenarios when you update your Bottlerocket nodes. You can update a managed node group to the latest Bottlerocket AMI release of the same Kubernetes version that’s currently deployed on the nodes with the following command. For example, you are upgrading to patch version 1.20.7.

eksctl upgrade nodegroup --name=bottlerocket-nodegroup --cluster=bottlerocket-cluster

You can upgrade a node group to the same version as the control plane’s Kubernetes version. For example, if you have a cluster running Kubernetes 1.21, you can upgrade nodes currently running Kubernetes 1.20 to version 1.21 with the following command.

eksctl upgrade nodegroup --name=bottlerocket-nodegroup --cluster=bottlerocket-cluster --kubernetes-version=1.21

You can also upgrade to a specific AMI release version instead of the latest version.

eksctl upgrade nodegroup --name=bottlerocket-nodegroup --cluster=bottlerocket-cluster --release-version=1.21

Key considerations

Bottlerocket is built from the ground up with only the minimum components necessary to run containers installed on the host. Any additional software, such as monitoring agents or metric collection systems like Container Insights, Prometheus, or Open Telemetry, must be run as Daemonsets. Bottlerocket also recommends using static pods or host containers.

You can use launch templates with Bottlerocket for scenarios such as additional arguments to kubelet, or to fully control volume type and size. When using a launch template with a Bottlerocket managed node group, user data must be in TOML format. Refer to the Bottlerocket documentation for a complete list of Kubernetes settings allowed via user data.

Bottlerocket is image-based and does not include a package manager for customization at runtime. One of the main design goals is to keep the OS image as small as possible for security and performance reasons. Different variants are available for use with different orchestrators and versions, each with its own specific set of software and API settings. A variant is essentially a list of packages installed, plus a model that defines the API. The Kubernetes variant includes kubelet, Container Network Interface (CNI) plugins, and AWS IAM authenticator needed to run a Kubernetes node on AWS.

We encourage you to use Daemonsets rather than building a custom Bottlerocket variant. For specific use cases not supported by Bottlerocket AMIs, you can use the custom AMI feature of managed node groups. When using a custom AMI, you will need to include the launch template with required bootstrap data (Kubernetes cluster connect) in EC2 user data. You will manage the custom AMI, launch template, and user data. Please refer to the Amazon EKS user guide for additional information about using custom AMIs.

Additionally, if you want to modify the host, we recommend that you use bootstrap containers for Bottlerocket. Bootstrap containers are host containers that can be used to initialize the host before the launch of services such as Kubernetes. Bootstrap containers are quite similar to standard host containers (control and admin); they have persistent storage and the ability to store optional user data. Bootstrap containers have access to the host’s root filesystem and all devices, and are configured with the CAP_SYS_ADMIN capability. This allows bootstrap containers to generate visible files, directories, and mounts on the host. Before the boot script is executed, the bootstrap containers are executed. You can utilize the launch template and user data to configure bootstrap containers by following the procedures outlined here. Please refer to the Amazon EKS user guide for additional information on how to use launch templates with Bottlerocket nodes.

Cleanup

Once you’ve completed the demonstration, you can delete the resources created to avoid future costs.

kubectl delete svc sample-app
kubectl delete deployment sample-app
eksctl delete nodegroup --cluster bottlerocket-cluster --name bottlerocket-nodegroup
eksctl delete cluster --name=bottlerocket-cluster

Conclusion

In this post, we showed how to use Bottlerocket natively with Amazon EKS managed node groups and how you can interact directly with the Bottlerocket cluster nodes. The components of Bottlerocket are open source. Bottlerocket is intended to be a collaborative community project, so you may contribute directly and create your own customized versions. You can get involved with the Bottlerocket GitHub project, and participate by opening issues and creating pull requests.

We are happy to introduce Amazon EKS native managed node group support for Bottlerocket. We have an exciting roadmap ahead, including in-place updates for managed node groups and Bottlerocket support for accelerated instance types in Amazon EC2. To learn about upcoming features, or to make suggestions, please visit the AWS Containers Roadmap or the Bottlerocket roadmap on GitHub.