Managing Pod Security on Amazon EKS with Kyverno

This blog post was co-written by Jim Bugwadia, Co-founder and CEO, Nirmata.

This post is a follow-up to our previous blog, Implementing Pod Security Standards in Amazon EKS.

Introduction

Pods are the fundamental unit of execution in Kubernetes. In this post, we’ll look at how Kyverno can be used to manage pod security for Amazon Elastic Kubernetes Service (Amazon EKS) with the new Kubernetes Pod Security Admission and Pod Security Standards features.

Pod security admission

The Kubernetes Pod Security Admission (PSA) feature went Beta in Kubernetes 1.23 and stable in 1.25. It implements a set of security controls defined in the Pod Security Standards (PSS). PSS defines 17 security controls for pod configurations organized into three levels: Privileged (unsecure), Baseline (secure), and Restricted (highly secure). PSA is a Kubernetes feature that provides an implementation of the PSS. The implementation consists of an admission controller that validates admission of pods against one of the three security levels, for each namespace, which is usually based on a static configuration file and namespace labels. The PSA uses three modes of operation:

enforce: Policy violations will cause the pod to be rejected.
audit: Policy violations trigger the addition of an audit annotation to the event recorded in the audit log, but are otherwise allowed.
warn: Policy violations will trigger a user-facing warning, but are otherwise allowed.

The Kubernetes API server static configuration provides the ability to define additional PSA/PSS settings, such as exemptions based on usernames, runtime classes, and namespaces. Available (Beta) as of Amazon EKS 1.23, the PSA/PSS implementation doesn’t include the ability to customize the static configuration of PSA/PSS. This is a conscious decision to reduce Kubernetes API sever complexity and stay with default settings.

Note: In order to ensure operational stability and maintain our SLA, Amazon EKS doesn’t allow custom configuration of API server settings. Amazon EKS evaluates requests for custom configurations on a case-by-case basis, and may allow a setting to be configured if there is enough customer demand. As an example, the Amazon EKS API supports OpenID Connect (OIDC) user authentication, which is then passed as API server flags when a cluster is started. Customers can use the Containers Roadmap to open and monitor issues, and add their influence to existing issues.

The default configuration used by Amazon EKS 1.23 and beyond enables the Privileged PSS profile, utilizing the latest version of PSA modes, for all pods in a cluster, with no configured exemptions.

...
    defaults:
      enforce: "privileged"
      enforce-version: "latest"
      audit: "privileged"
      audit-version: "latest"
      warn: "privileged"
      warn-version: "latest"
    exemptions:
      # Array of authenticated usernames to exempt.
      usernames: []
      # Array of runtime class names to exempt.
      runtimeClasses: []
      # Array of namespaces to exempt.
      namespaces: []
...

With this model, Kubernetes Namespaces in Amazon EKS 1.23 and beyond can opt in to more restrictive pod security settings, using the PSA/PSS labels.

# The per-mode level label indicates which policy level to apply for the mode.
#
# MODE must be one of `enforce`, `audit`, or `warn`.
# LEVEL must be one of `privileged`, `baseline`, or `restricted`.
pod-security.kubernetes.io/<MODE>: <LEVEL>

# Optional: per-mode version label that can be used to pin the policy to the
# version that shipped with a given Kubernetes minor version (for example v1.24).
#
# MODE must be one of `enforce`, `audit`, or `warn`.
# VERSION must be a valid Kubernetes minor version, or `latest`.
pod-security.kubernetes.io/<MODE>-version: <VERSION>

While PSA is built in and enabled by default in Amazon EKS 1.23 and beyond, it’s also configured to use the Privileged PSS level by default. This means that there is no security enforced by default, unless namespace labels are used to opt in to more secure settings; pods can be executed with root privileges by default.

Another common challenge with PSA is that the configured security levels apply to all pods within a namespace. The rationale for this is that all pods in a namespace can access all service accounts and other sensitive resources in the namespace. However, using granular namespaces isn’t always practical and there is often a need to provide a policy exception to a single pod or container within a namespace, while restricting access for other pods or containers in the same namespace. Using Kyverno, users can enforce additional behaviors that complement the PSA/PSS settings, increase pod security, and provide a better user experience for Amazon EKS.

Kyverno

Kyverno, a Cloud Native Computing Foundation (CNCF) incubating project, is a Policy-as-Code (PaC) solution that includes a policy engine designed for Kubernetes. The Kyverno policy engine is installed into Kubernetes clusters and integrated to the Kubernetes API server as Dynamic Admission Controllers. This allows Kyverno policies to be used to mutate and validate inbound Kubernetes API server requests before the requests persist changes into the internal Kubernetes etcd data store.

Note: Kubernetes Dynamic Admission Controllers use webhook integration to call validating and mutating services. These calls have timeout settings (default 10 seconds) for how long the call will wait before a timeout occurs. The webhook configurations also include a failurePolicy setting to configure how the API server should respond when the webhook call doesn’t return within the configured timeout period. The webhook will either fail open (where the API server request is allowed to proceed) or fail closed (where the API server request is blocked). There are tradeoffs for each failure scenario. While a fail-open scenario could be seen as a potential security issue, a fail-closed scenario could cause operational issues for the cluster. For additional information about these scenarios and how they are configured in Kyverno, please see the Kyverno Security vs. Operability documentation.

Kyverno policies are declarative Kubernetes resources, written in YAML, with no new policy language to learn. Policy results are also available as Kubernetes resources and as events. Kyverno policies can be used to validate, mutate, and generate resource configurations, and validate image signatures and attestations.

Installing Kyverno

To exercise the new pod security functionality provided by Kyverno, we must install the Kyverno policy engine and associated configurations into an existing Amazon EKS cluster. You can install the latest version of Kyverno using this Helm command:

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno --namespace kyverno kyverno/kyverno --create-namespace

Since Kyverno is built for Kubernetes, it can be used to complement and enhance the Kubernetes PSA feature. In the following sections of this post, we’ll discuss ways you can use Kyverno to apply PSS settings to your Amazon EKS clusters.

Enforcing pod security labels

As mentioned earlier, with the default PSA/PSS settings in Amazon EKS, namespaces must opt in to more restrictive pod security with PSA/PSS labels. You can use Kyverno to help manage pod security by enforcing the use of these pod security labels. The following Kyverno policy mutates inbound Kubernetes API server requests that create namespaces, and adds the pod security enforce mode label, if one is not already defined.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-pss-labels    
spec:
  rules:
  - name: add-restricted
    match:
      resources:
        kinds:
        - Namespace
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(pod-security.kubernetes.io/enforce): restricted

The result of this mutation is a namespace correctly labeled to the use the PSA enforce mode with the PSS Restricted level. All pods running in namespaces mutated by this policy will have the tightest pod security available from the PSA/PSS settings. To update existing namespaces, you can refer to this sample mutation policy to add privileged labels to existing namespaces.

After mutation, it’s still a good zero-trust practice to validate inbound Kubernetes API server requests, ensuring that the desired settings are present in the request payload, and preventing requests from making unwanted cluster changes. To that end, the following Kyverno validate policy rule can restrict the value of the PSA enforce mode to the PSS Baseline or Restricted levels.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: validate-pss-labels
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: check-restricted
    match:
      resources:
        kinds:
        - Namespace
    validate:
      message: "the privileged pod security level is not allowed"
      pattern:
        metadata:
          labels:
            pod-security.kubernetes.io/enforce: "baseline | restricted"

With these policies in place, PSS is enforced on all new namespaces at either the Baseline or Restricted levels, providing an improved security posture over the default PSA settings.

Managing pod security levels and exceptions

In Kyverno release 1.8, a new validation rule type validate.podSecurity was introduced. This declaration directly uses the in-tree Kubernetes PSS implementation while allowing more flexibility. Here are some of the major benefits:

Cluster-wide pod security defaults do not require updating the static configuration file
Namespace level pod security application does not require labels
Pod security checks are automatically applied to pod controllers (e.g., deployments)
Granular exemptions can be configured at a container image level
Violations are reported in an in-cluster report
Static validation and testing can be performed via the Kyverno command line interface (CLI) without requiring a cluster

The following example cluster policy applies the Restricted PSS level to all namespaces except for the kube-system namespace. Within each namespace, except kube-system, all pods have the same pod security settings, which is set to PSS Restricted level. This validation policy is used as part of a Defense-in-Depth (DiD) strategy that augments the prior namespace mutation and validation policies.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: psa
spec:
  background: true
  validationFailureAction: enforce
  rules:
  - name: restricted
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      all:
      - resources:
          namespaces:
          - kube-system
    validate:
      podSecurity:
        level: restricted
        version: latest

Existing namespace labels are ignored

An important point to remember about the new Kyverno validate.podSecurity rule is that it neither requires nor considers existing PSA/PSS namespace labels. For example, given the namespace configuration below, that already includes PSA/PSS labels for the Privileged PSS level, the above Kyverno policy prevents a pod from being created, if a combination of the securityContext elements at the pod and container levels violate the Restricted PSS controls.

apiVersion: v1
kind: Namespace
metadata:
  name: policy-test
  labels: 
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged

In this scenario, the Kyverno policy using the validate.podSecurity rule overrides the PSA/PSS namespace labels.

Additional granularity with pod specific exemptions

As mentioned and demonstrated above, the namespace opt-in model to PSA/PSS pod security can be specified at the namespace level, with each pod in its respective namespace getting the same applied pod security. However, in the case where you need different PSA/PSS settings between pods in the same namespace, Kyverno policies can be used to provide that granularity.

Applying pod-specific exemptions to namespace PSA/PSS settings provides sub-namespace granularity. The following Kyverno policy applies the Restricted PSS level and configures an exemption for pods using the nginx:1.23.1-alpine image in the exempt namespace. All other pods in the same namespace, or in any other namespace, are subjected to the Restricted PSS level.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: exempt-run-as-non-root
spec:
  validationFailureAction: enforce
  applyRules: One
  rules:
  - name: exempt-run-as-non-root
    match:
      all:
      - resources:
          kinds:
          - Pod
          namespaces:
          - exempt
    validate:
      podSecurity:
         level: restricted
         exclude:
          - controlName: "Running as Non-root"
            images:
            - nginx:1.23.1-alpine
  - name: restricted
    match:
      all:
      - resources:
          kinds:
          - Pod
    validate:
      podSecurity:
         level: restricted

Rules processing – Controlling how many rules are applied

In the above policy there are two defined rules: exempt-run-as-non-root and restricted. The use of the applyRules: One directive in the above policy instructs Kyverno to stop processing additional policy rules after the first matching rule is applied. Otherwise, the default behavior is to process all rules declared in a policy. In this example, this directive is used to create an exception that applies to a single workload using the nginx:1.23.1-alpine image in the exempt namespace, while providing a default rule that applies to all other pods, regardless of namespace.

With the 2-rule policy above, the flow (depicted below) is achieved.

A request is submitted to the Kubernetes API server to create a pod
- If the namespace in the request is not the exempt namespace, then the Restricted PSS security profile, with the enforce PSA mode, is applied to the pod
- If the namespace in the request is the exempt namespace then check the container image in the pod
  - If the container image in the pod is nginx:1.23.1-alpine, don’t apply a PSA/PSS combination

Auto-generated policies

PSA operates at the pod level, but in practice pods are usually managed by pod controllers, like Deployments. Having no indication of pod security errors at the pod controller level makes issues complex to troubleshoot. The PSA enforce mode is the only PSA mode that stops pods from being created; however, PSA enforce doesn’t act on controller resources that create pods. To help this user experience, we recommend that PSA warn and audit modes are also used with enforce; PSA warn and audit modes will at least indicate that pod-creating controller resources are trying to create pods that would otherwise fail the applied PSS level. This user experience can present a challenge to adopters of the PSA/PSS features.

Using PaC solutions with Kubernetes presents another challenge: writing and maintaining policies to cover all the different Kubernetes resources used within clusters. With the Kyverno Auto-Gen Rules for Pod Controllers feature, the above pod policies auto-generate associated pod controller (Deployment, DaemonSet, etc.) policies. This Kyverno feature increases the expressive nature of policies, and reduces the effort to maintain policies for associated resources. These additional auto-generated policies also improve the above-mentioned PSA user experience where pod-creating resources are not prevented from progressing, while the underlying pods are prevented.

As seen below, the auto-generated controller policies are indicated in the status field of the original pod policy.

...
status:
  autogen:
    rules:
    - exclude:
        all:
        - resources:
            namespaces:
            - kube-system
        resources: {}
      generate:
        clone: {}
        cloneList: {}
      match:
        any:
        - resources:
            kinds:
            - DaemonSet
            - Deployment
            - Job
            - StatefulSet
        resources: {}
      mutate: {}
      name: autogen-restricted
      validate:
        podSecurity:
          level: restricted
          version: latest
    - exclude:
        all:
        - resources:
            namespaces:
            - kube-system
        resources: {}
      generate:
        clone: {}
        cloneList: {}
      match:
        any:
        - resources:
            kinds:
            - CronJob
        resources: {}
      mutate: {}
      name: autogen-cronjob-restricted
      validate:
        podSecurity:
          level: restricted
          version: latest
...

Note: The Kyverno project provides an extensive policy library hosted on its project site, including specific policies related to pod security.

User-friendly reporting

Kyverno includes policy reporting, using the open format defined by the Kubernetes Policy Working Group. Kyverno emits these policy reports when admission actions (CREATE, UPDATE, DELETE) are performed. These reports are also generated as a result of background scans that apply policies on existing resources. The reports can be found using the following kubectl command:

kubectl -n kyverno get policyreports.wgpolicyk8s.io

NAME       PASS   FAIL   WARN   ERROR   SKIP   AGE
cpol-psa   2      0      0      0       0      4h19m

In the output above, the psa cluster policy indicates two passes. The entire report can be seen below.

kubectl -n kyverno get policyreports.wgpolicyk8s.io

NAME       PASS   FAIL   WARN   ERROR   SKIP   AGE
cpol-psa   2      0      0      0       0      4h19m
In the output above, the psa cluster policy indicates two passes. The entire report can be seen below.
kubectl -n kyverno get policyreports.wgpolicyk8s.io cpol-psa -o yaml

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
  creationTimestamp: "2022-11-01T13:03:23Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kyverno
  name: cpol-psa
  namespace: kyverno
  resourceVersion: "3787"
  uid: eec4d26f-c276-47dc-ae8f-033ba4dc1dcb
results:
- message: Validation rule 'autogen-restricted' passed.
  policy: psa
  resources:
  - apiVersion: apps/v1
    kind: Deployment
    name: kyverno
    namespace: kyverno
    uid: c6635d26-5abb-4e2c-8d2a-802c72eeb777
  result: pass
  rule: autogen-restricted
  scored: true
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1667307788
- message: Validation rule 'restricted' passed.
  policy: psa
  resources:
  - apiVersion: v1
    kind: Pod
    name: kyverno-6d488fd94b-2f42v
    namespace: kyverno
    uid: ac769e05-84ab-40ea-a939-c4802aa689bf
  result: pass
  rule: restricted
  scored: true
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1667307788
summary:
  error: 0
  fail: 0
  pass: 2
  skip: 0
  warn: 0

Kyverno CLI shifts policy evaluation to the left

It’s not always optimal to test policy evaluation in Kubernetes clusters. It’s cheaper (time and money) to catch potential policy issues (violations, errors, etc.) upstream of Kubernetes in an automated DevOps pipeline. To shift policy testing to the left (for less cost and overhead), before we touch our clusters, we can use the Kyverno command-line interface (CLI) to apply policies to Kubernetes resource YAML files.

There are multiple ways to install the Kyverno CLI. For example, it can be installed as a kubectl plugin, using krew:

kubectl krew install kyverno

Once the Kyverno CLI is installed, the following command can be used to apply a Kyverno policy to a Kubernetes resource YAML file (no Kubernetes cluster needed):

kubectl kyverno apply policies/pod-restricted.yaml --resource=tests/2-dep-sec-cont.yaml

Applying 1 policy rule to 1 resource...

policy psa -> resource policy-test/Deployment/test failed:
1. autogen-restricted: Validation rule 'autogen-restricted' failed. It violates PodSecurity "restricted:latest": ({Allowed:false ForbiddenReason:allowPrivilegeEscalation != false ForbiddenDetail:container "test" must set securityContext.allowPrivilegeEscalation=false})
({Allowed:false ForbiddenReason:allowPrivilegeEscalation != false ForbiddenDetail:container "test" must set securityContext.allowPrivilegeEscalation=false})
({Allowed:false ForbiddenReason:unrestricted capabilities ForbiddenDetail:container "test" must set securityContext.capabilities.drop=["ALL"]})
({Allowed:false ForbiddenReason:unrestricted capabilities ForbiddenDetail:container "test" must set securityContext.capabilities.drop=["ALL"]})
({Allowed:false ForbiddenReason:runAsNonRoot != true ForbiddenDetail:pod or container "test" must set securityContext.runAsNonRoot=true})
({Allowed:false ForbiddenReason:seccompProfile ForbiddenDetail:pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"})
({Allowed:false ForbiddenReason:seccompProfile ForbiddenDetail:pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"})


pass: 0, fail: 1, warn: 0, error: 0, skip: 2

The above command evaluated a Kubernetes Deployment resource YAML file against an example pod policy, which was mentioned earlier in this post. Even in the Kyverno CLI, the auto-generate feature created the Deployment policy, based on the pod policy, inline to the policy evaluation.

Cleaning up

To remove Kyverno from your cluster, you can follow the uninstall instructions. The helm uninstall command is seen below:

helm uninstall kyverno kyverno/kyverno --namespace kyverno

As mentioned in the Kyverno documentation, Kyverno will try to remove all its webhook configurations. This can be done manually or as a final step with the following kubectl command:

kubectl delete mutatingwebhookconfigurations kyverno-policy-mutating-webhook-cfg kyverno-resource-mutating-webhook-cfg kyverno-verify-mutating-webhook-cfg
kubectl delete validatingwebhookconfigurations kyverno-policy-validating-webhook-cfg kyverno-resource-validating-webhook-cfg

Conclusion

In this post, we showed you how to augment the Kubernetes PSA/PSS configurations with Kyverno. Pod Security Standards (PSS) and the in-tree Kubernetes implementation of these standards, Pod Security Admission (PSA), provide good building blocks for managing pod security. The majority of users switching from Kubernetes Pod Security Policies (PSP) should be successful using the PSA/PSS features.

Kyverno augments the user experience created by PSA/PSS, by leveraging the in-tree Kubernetes pod security implementation, and providing several helpful enhancements for operationalization. You can use Kyverno to govern the proper use of pod security labels. In addition, you can use the new Kyverno validate.podSecurity rule to easily manage pod security standards with additional flexibility and an enhanced user experience. And, with the Kyverno CLI, you can automate policy evaluation, upstream of your clusters.

Learn more about Kyverno at https://kyverno.io.

Containers