Provisioning and Securing Bottlerocket OS-Based Amazon EKS Clusters Using Nirmata Kubernetes Platform

By Ritesh Patel, Co-Founder and VP Products – Nirmata
By Mikhail Shapirov, Sr. Partner Solutions Architect, Containers – AWS

Nirmata

Organizations of all sizes and across all industries are adopting cloud-native technologies to take advantage of the agility, flexibility, and resilience they offer. Containers have become the de facto way to package applications, and Kubernetes is the orchestration platform of choice.

Unfortunately, with the speed and agility of distributed cloud-native applications comes complexity and risk. Although Kubernetes is extremely powerful and versatile, the configuration and operation of these environments quickly becomes difficult to manage and scale.

Securing the Kubernetes environment without compromising on developer agility holds the key to success with cloud-native adoption. Some of the challenges that enterprise customers need to overcome when adopting Kubernetes include:

• Selecting a secure and performant operating system (OS): OS defines performance and capabilities available in containers, and the initial surface of security risks that must be protected.
• Configuration management: Although the declarative nature of Kubernetes is a powerful mechanism that directly enables self-healing, it also increases the amount of configuration that has to be managed.
• Segregation of responsibilities: Another challenge is determining whose responsibility it is to configure the right settings for security, best practices, and standardization. The solution to address this challenge is a policy engine such as Kyverno, which uses policies to validate configurations for best practices and security compliance, and also automatically modifies and generates additional configurations when needed.
• Self-service: The desired developer agility with cloud-native environments cannot be achieved without enabling self-service. Developers want on-demand access to Kubernetes clusters, but operators want to ensure the clusters are provisioned with the requisite security: a secure host OS, policy engine for governance and compliance, and possibly other runtime security tools.

The Nirmata Kubernetes Platform is designed to address these challenges comprehensively. It’s an open and easy-to-adopt platform to deploy, operate, and optimize Kubernetes workloads on any infrastructure, enabling self-service, segregation of responsibilities, and security and governance controls.

In this post, we’ll discuss how secure developer agility can be achieved using Bottlerocket OS and Kyverno on Amazon Elastic Kubernetes Service (Amazon EKS) managed by Nirmata Kubernetes Platform.

Nirmata is an AWS Partner and container-native microservice platform that enables enterprise software development teams to deliver complex, distributed applications across public and private cloud.

Bottlerocket OS

Bottlerocket is an open source Linux-based operating system that is purpose-built by Amazon Web Services (AWS) for running containers. It focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads.

To accomplish this, Bottlerocket simplifies the OS by including only the essential software needed to run containers. This is important for a couple of reasons. First, it reduces the footprint of the underlying OS, aiding overall performance. Second, it limits the exposure to potential attacks, as many of the tools used by would-be attackers simply aren’t there.

Bottlerocket was designed to be performant, secure, and easy to maintain. It contains the following primary components:

• Minimal OS that includes the Linux kernel (5.4), system software, and containerd as the container runtime.
• Atomic update mechanism to apply and rollback OS updates in a single step (also known as transactional updates).
• Integrations with container orchestrators such as Amazon EKS to manage and orchestrate updates.
• Admin container that can be optionally run for advanced troubleshooting and debugging.

When used with Amazon EKS, customers can take advantage of reliable security updates. The mechanism used by Bottlerocket enables automatic software patching on Amazon EKS and Amazon EKS-Anywhere by the Bottlerocket Kubernetes Operator on self-managed nodes. This aligns with the standard update processes when used with EKS Managed Node Groups as a custom Amazon Machine Image (AMI).

When selecting an OS for worker nodes of your EKS cluster, it makes sense to take advantage of an option that is purpose-built to run containers. The advantages for Amazon EKS are not limited to compactness and security features; it also provides customers with an option to use the power of the 5.x Linux kernel, with performance benefits and eBPF technology.

Kyverno Policy Engine

Kyverno is an open source Kubernetes-native policy engine that runs as an admission controller and can validate, mutate, and generate any configuration data based on customizable policies.

Although other general purpose policy solutions were retrofitted to Kubernetes, Kyverno was designed ground-up for Kubernetes. Like Kubernetes, Kyverno adopts a declarative management paradigm. Kyverno policies are simply Kubernetes resources, and don’t require learning a new language. Kyverno helps secure the Kubernetes configuration by preventing misconfigurations and enhancing security.

Nirmata Kubernetes Platform

Nirmata Kubernetes Platform integrates the required tools and processes, enabling enterprises to standardize on Kubernetes as their cloud-native operating system, cleanly decouple applications from infrastructure, and use ongoing open source innovation.

The platform helps enterprise operations teams deliver secure self-service for developers unlocking DevOps agility. Nirmata Kubernetes Platform integrates with Amazon EKS and supports provisioning of Bottlerocket OS-based EKS clusters. It also includes native support for the Kyverno policy engine, ensuring the provisioned clusters are protected with advanced security and governance features.

Putting it All Together

Next, we’ll show you how to take advantage of the advanced security features when provisioning Kubernetes clusters on Amazon EKS using the Nirmata Kubernetes Platform with Bottlerocket OS and Kyverno policy engine.

Creating the Cluster Type

First, let’s create a cluster type for Amazon EKS. A cluster type lets you specify all of the necessary configuration to provision the cluster. After the cluster type is created, you can give developers access to use the cluster type so they can provision clusters based on predefined configurations.

In the top left corner of the Nirmata console, select the menu icon and then select Clusters > Cluster Types.

Choose Add Cluster Type, and for Cloud Provider Managed Clusters you can select EKS.

Figure 1 – Defining EKS cluster type.

Now, specify the name for the cluster type and select the configuration for your cluster control plane. When configuring the cluster node pool, select Use Custom AMI and then Amazon EKS-optimized Bottlerocket AMI in the Image ID field.

You’ll notice the user data will be automatically filled. User data can be used to initialize the nodes for your cluster. For Bottlerocket AMI, the user data is specified in TOML format.

Figure 2 – Selecting Bottlerocket AMI for the managed node group.

Also, when creating the cluster type, you can specify the add-ons you would like to be deployed on your cluster. By default, Kyverno will be selected as an add-on.

You can configure access control on the cluster type so only the teams allowed to use the cluster type can create clusters with it.

Configuring Policy Groups

To improve the security of your cluster, you can specify the policies that should be deployed to the cluster after it’s created. By default, there are three policy groups created for each account.

You can add more policy groups and configure them as “default” policy groups so they get deployed to newly-created clusters. Policy groups contain a set of policies that ensure the cluster uses security and compliance features according to the bar defined by the policy group. These policies are enforced in the cluster by Kyverno.

Policy groups are an effective way to address regulatory compliance, like PCI compliance or cross-cutting concerns such as multi tenancy. With policy groups, customers don’t need to understand nuances of each individual policy applied.

For example, multi tenancy requires network isolation per namespace and clear controls over ingress/egress rules. It also requires security controls on namespace access with segregation of responsibilities between the platform teams (admin-level access) and tenant team (could be a development team or an actual customer tenant).

The “Multi Tenancy” policy group will address these rules and provide the basis to extend them based on the customer domain to include additional custom controls, if necessary.

Figure 3 – Managed policy groups.

Creating a Cluster

To create a new cluster, in the top left corner of the Nirmata console, select the menu icon and then select Clusters > Clusters.

Choose Add Cluster, and in the Cloud Provider Managed Clusters panel you can choose EKS.

Figure 4 – Creating an Amazon EKS cluster.

Within a few minutes, the cluster should be created and the add-ons will be deployed. In addition to Kyverno, the policies that are configured in your policy groups will also be deployed.

Deploying an Insecure Application

After the cluster is up and running, you can start deploying applications. You can now verify the policies that are applied to the cluster by creating a deployment with an insecure configuration (for example, run as root is enabled).

Let’s deploy nginx-insecure.yaml which allows the container to run under the root user:

securityContext:
  runAsNonRoot: false  

The deployment will not be allowed and you’ll receive an error message similar to the following:

$ demo % kubectl create -f nginx-secure.yaml

Error from server: error when creating "nginx-insecure.yaml": admission webhook "validate.kyverno.svc" denied the request:
resource Deployment/default/nginx was blocked due to the following policies
require-run-as-non-root:
  autogen-check-containers: 'validation error: Running as root is not allowed. The
    fields spec.securityContext.runAsNonRoot, spec.containers[*].securityContext.runAsNonRoot,
    and spec.initContainers[*].securityContext.runAsNonRoot must be `true`. Rule autogen-check-containers[0]
    failed at path /spec/template/spec/securityContext/runAsNonRoot/. Rule autogen-check-containers[1]
    failed at path /spec/template/spec/containers/0/securityContext/.'

Deploying a Secure Application

Let’s fix the application and deploy it to the cluster.

nginx-secure.yaml now has a Kubernetes deployment with the following securityContext:

securityContext:
  runAsNonRoot: true

Let’s deploy it in the cluster protected by our policy:

demo % kubectl create -f nginx-secure.yaml

deployment.apps/nginx created

This time it works as expected, and the pods are up and running.

Conclusion

As demonstrated in this post, you don’t need to trade off security and compliance to achieve developer agility. Amazon EKS is an efficient and versatile service for the provisioning of managed Kubernetes clusters.

With the introduction of Bottlerocket OS, the cluster nodes are now more secure and deliver better performance. Applying and enforcing policies using Kyverno also helps ensure configuration security.

Finally, using the Nirmata Kubernetes Platform, you can enable developer self-service for Kubernetes clusters without much effort and ensure the applications deployed to these clusters are generally secure.

To learn more about Kyverno, join the large and rapidly growing community and discover more in the Nirmata blog series. You can try out this capability with a free trial of Nirmata Kubernetes Platform.

References

• Kyverno GitHub page
• Kyverno Sample Policies
• Kyverno Community Resources

.

.

Nirmata – AWS Partner Spotlight

Nirmata is an AWS Partner that enables enterprise software development teams to deliver complex, distributed applications across public and private cloud.

Contact Nirmata| Partner Overview | AWS Marketplace

*Already worked with Nirmata? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.