Amazon EKS add-ons: Advanced configuration

This post is a follow-up to our previous post, Amazon EKS add-ons preserve customer edits.

Introduction

In October 2022, the Amazon Elastic Kubernetes Service (Amazon EKS) add-ons team introduced the ability to preserve edits, enabling customers to safely modify the configuration of Amazon EKS add-ons by using the Kubernetes application programming interface (API). This enhancement enabled users to preserve add-on configuration changes on their clusters during update operations by passing in the --resolve-conflicts PRESERVE parameter.

Today, we’re expanding on this work to give customers more control over their operational software, with the launch of advanced configuration support for Amazon EKS add-ons. Customers can now supply their configuration directly through the Amazon EKS add-ons API, to install and configure their operational software during cluster creation in a single step. This enables customers to leverage their existing Infrastructure as Code (IaC) tools to configure Amazon EKS vended operational software and meet performance, compliance, or additional requirements not handled by default configuration settings. Support for advanced configuration through the Amazon EKS add-ons API is available today for the following add-ons:

Configuration parameters available for these add-ons address some of the most highly requested issues on the containers roadmap, including: modifying the CoreDNS corefile, simplifying the configuration for running CoreDNS on AWS Fargate, enabling IPVS mode with kube-proxy, increasing kube-proxy memory, and enabling VPC CNI parameters such as prefix delegation.

Configuration support also unlocks an exciting new use case of simplifying the level of input required to get started with certain operational software on Amazon EKS. A good example is the AWS Distro for OpenTelemetry (ADOT), the AWS vended and support distribution of OpenTelemetry that provides open-source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. Using Amazon EKS add-ons, you install the OpenTelemetry Operator, and then configure an OpenTelemetry Collector instance through the OpenTelemetryCollector CustomResourceDefinition (CRD). This CRD supports the full surface area of the OpenTelemetry Collector, but this can be an overwhelming amount of information for users getting started with observability on their clusters. With Amazon EKS add-ons configuration support for ADOT, you can now optionally specify cloudwatch: true, or amp:true in your configuration input, and Amazon EKS automatically applies a best practice custom resource to configure an OpenTelemetry Collector to send observability data to your desired destination, such as Amazon CloudWatch or Amazon Managed Service for Prometheus.

Feature overview

A new configurationValues parameter has been added to the Amazon EKS add-ons API which accepts configuration as a JSON or a YAML blob. The configuration blob needs to conform to the underlying add-on configuration JSON schema, which is now available through the new Amazon EKS DescribeAddonConfiguration API. As seen below, the configurationValues parameter can be used with the aws eks create-addon and aws eks update-addon AWS Command Line Interface (AWS CLI) command.

Example create add-on workflow

To illustrate how customers can use the new configurationValues JSON parameter, let’s consider the following scenario. Given an existing Amazon EKS cluster, we want to use the AWS CLI to add the VPC CNI add-on to the existing cluster with the create-addon command. However, we want to modify the default settings and increase the number of available IP addresses on our Amazon EC2 nodes by enabling Prefix Delegation. To customize the add-on during creation, we must first discover if the desired configuration item is available for the version of the add-on we plan to use. We’ll use the new Amazon EKS DescribeAddonConfiguration API, with the corresponding describe-addon-configuration CLI command to perform this discovery.

In this scenario, discovery is a two-step process. First, we use the describe-addon-versions CLI command, seen in the following example, to retrieve the versions from the Amazon EKS API.

aws eks describe-addon-versions --addon-name vpc-cni

{
    "addons": [
        {
            "addonName": "vpc-cni",
            "type": "networking",
            "addonVersions": [
                {
                    "addonVersion": "v1.12.0-eksbuild.1",
                    "architecture": [
                        "amd64",
                        "arm64"
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.24",
                            "platformVersions": [
                                "*"
                            ],
                            "defaultVersion": false
                        },
...

As seen in the preceding output, versions of the add-on (i.e., VPC CNI in this case) are listed with corresponding architecture and cluster version compatibilities. With this information, we select the most appropriate version for our use case. Once the add-on version is selected, then we can use the new describe-addon-configuration CLI command to review the configuration items that can be configured.

aws eks describe-addon-configuration --addon-name vpc-cni --addon-version v1.12.0-eksbuild.1

{
        "addonName": "vpc-cni",
        "addonVersion": "v1.12.0-eksbuild.1",
        "configurationSchema": "{\"$ref\":\"#/definitions/VpcCni\",\"$schema\":\"http://json-schema.org/draft-06/schema#\",\"definitions\":{\"Cri\":{\"additionalProperties\":false,\"properties\":{\"hostPath\":{\"$ref\":\"#/definitions/HostPath\"}},\"title\":\"Cri\",\"type\":\"object\"},\"Env\":{\"additionalProperties\":false,\"properties\":{\"ADDITIONAL_ENI_TAGS\":{\"type\":\"string\"},\"AWS_VPC_CNI_NODE_PORT_SUPPORT\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_ENI_MTU\":{\"format\":\"integer\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_EXTERNALSNAT\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_LOGLEVEL\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_LOG_FILE\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_RANDOMIZESNAT\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_VETHPREFIX\":{\"type\":\"string\"},\"AWS_VPC_K8S_PLUGIN_LOG_FILE\":{\"type\":\"string\"},\"AWS_VPC_K8S_PLUGIN_LOG_LEVEL\":{\"type\":\"string\"},\"DISABLE_INTROSPECTION\":{\"format\":\"boolean\",\"type\":\"string\"},\"DISABLE_METRICS\":{\"format\":\"boolean\",\"type\":\"string\"},\"DISABLE_NETWORK_RESOURCE_PROVISIONING\":{\"format\":\"boolean\",\"type\":\"string\"},\"ENABLE_POD_ENI\":{\"format\":\"boolean\",\"type\":\"string\"},\"ENABLE_PREFIX_DELEGATION\":{\"format\":\"boolean\",\"type\":\"string\"},\"WARM_ENI_TARGET\":{\"format\":\"integer\",\"type\":\"string\"},\"WARM_PREFIX_TARGET\":{\"format\":\"integer\",\"type\":\"string\"}},\"title\":\"Env\",\"type\":\"object\"},\"HostPath\":{\"additionalProperties\":false,\"properties\":{\"path\":{\"type\":\"string\"}},\"title\":\"HostPath\",\"type\":\"object\"},\"Limits\":{\"additionalProperties\":false,\"properties\":{\"cpu\":{\"type\":\"string\"},\"memory\":{\"type\":\"string\"}},\"title\":\"Limits\",\"type\":\"object\"},\"Resources\":{\"additionalProperties\":false,\"properties\":{\"limits\":{\"$ref\":\"#/definitions/Limits\"},\"requests\":{\"$ref\":\"#/definitions/Limits\"}},\"title\":\"Resources\",\"type\":\"object\"},\"VpcCni\":{\"additionalProperties\":false,\"properties\":{\"cri\":{\"$ref\":\"#/definitions/Cri\"},\"env\":{\"$ref\":\"#/definitions/Env\"},\"resources\":{\"$ref\":\"#/definitions/Resources\"}},\"title\":\"VpcCni\",\"type\":\"object\"}}}"
    }

In the preceding output, the configurationSchema element contains the configurable elements for the selected version of the VPC CNI add-on.

Note: The returned configurationSchema is a standard JSON Schema, and each add-on and add-on version combination has its own configurationSchema. JSON schemas add metadata, such as descriptions, to individual fields, allowing for the schemas to serve as documentation for configuration options. Because JSON schema is an existing and established standard, it also has various third-party tools customers can use to view these schemas in a more user or developer friendly manner. Once such tool is the JSON Schema Viewer from Atlassian.

The configurable items in the configurationSchema field include environment variables and Kubernetes configurations, and are based on underlying Helm charts that are used by Amazon EKS to install, update, and uninstall add-on software to clusters. After reviewing the output, we see that among the schema elements is the ENABLE_PREFIX_DELEGATION environment variable, as seen in the following output.

{
        "$ref": "#/definitions/VpcCni",
        "$schema": "http://json-schema.org/draft-06/schema#",
        "definitions": {
            "Env": {
                "additionalProperties": false,
                "properties": {
    ...
                    "ENABLE_PREFIX_DELEGATION": {
                        "format": "boolean",
                        "type": "string"
                    },
    ...

The ENABLE_PREFIX_DELEGATION environment variable takes a string value formatted as a boolean (true/false). This variable defaults to false, as seen in the following example.

kubectl -n kube-system get ds aws-node -oyaml|grep ENABLE_PREFIX_DELEGATION -A1

- name: ENABLE_PREFIX_DELEGATION
  value: "false"

To more easily view the output of the describe-addon-configuration command, the output of the CLI command can be set to text, as seen in the following CLI command. This outputs unescaped JSON that’s easier to read and copy. The added query parameter does client-side filtering, and in this case, restricts the output to only return the configurationSchema field.

aws eks describe-addon-configuration --addon-name vpc-cni \
--addon-version v1.12.0-eksbuild.1 --query configurationSchema --output text

With this information, and the new configurationValues parameter, we can now craft the following create-addon CLI command, containing advanced configurations as custom YAML or JSON.

Note: The CLI command configuration-values parameter corresponds to the Amazon EKS add-on API configurationValues parameter.

# Configuration in YAML format
aws eks create-addon --cluster-name &lt;CLUSTER_NAME&gt; \
--addon-name vpc-cni --addon-version v1.12.0-eksbuild.1 \
--configuration-values $'env:\n ENABLE_PREFIX_DELEGATION: \"true\"' \
--resolve-conflicts OVERWRITE
{
    "addon": {
        "addonName": "vpc-cni",
        "clusterName": "<CLUSTER_NAME>",
        "status": "CREATING",
        "addonVersion": "v1.12.0-eksbuild.1",
        "health": {
            "issues": []
        },
...
        "configurationValues": "env:\n ENABLE_PREFIX_DELEGATION: \"true\""
    }
}

# Configuration in JSON format
aws eks create-addon --cluster-name &lt;CLUSTER_NAME&gt; \
--addon-name vpc-cni --addon-version v1.12.0-eksbuild.1 \
--configuration-values '{"env":{"ENABLE_PREFIX_DELEGATION":"true"}}' \
--resolve-conflicts OVERWRITE

{
    "addon": {
        "addonName": "vpc-cni",
        "clusterName": "<CLUSTER_NAME>",
        "status": "CREATING",
        "addonVersion": "v1.12.0-eksbuild.1",
        "health": {
            "issues": []
        },
...
        "configurationValues": "{\"env\":{\"ENABLE_PREFIX_DELEGATION\":\"true\"}}"
    }
}

The preceding output indicates that the add-on is in the CREATING status. We can use the describe-addon CLI command to check the status and confirm that the added advanced configuration is now part of the ACTIVE add-on.

aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name vpc-cni

{
        "addon": {
            "addonName": "vpc-cni",
            "clusterName": "<CLUSTER_NAME>",
            "status": "ACTIVE",
            "addonVersion": "v1.12.0-eksbuild.1",
            "health": {
                "issues": []
            },
    ...
            "configurationValues": "env:\n ENABLE_PREFIX_DELEGATION: \"true\""
        }
    }

With the following kubectl command we can also verify that the aws-node Daemonset in the target cluster received the specified advanced configuration.

kubectl -n kube-system get ds aws-node -oyaml|grep ENABLE_PREFIX_DELEGATION -A1

        - name: ENABLE_PREFIX_DELEGATION
          value: "true"

YAML or JSON input file

The CLI command configuration-values parameter accepts a file path instead of literal YAML or JSON text, as seen in the following example.

YAML file input
# CLI command input as file (snippet)
...
--configuration-values 'file://example.yaml'
...

# YAML input file
env:
  ENABLE_PREFIX_DELEGATION: 'true'

JSON file input
# CLI command input as file (snippet)
...
--configuration-values 'file://example.json'
...

# JSON input file
{
    "env": {
        "ENABLE_PREFIX_DELEGATION": "true"
    }
}

# CLI command output (snippet)
...
"configurationValues": "{\n    \"env\": {\n        \"ENABLE_PREFIX_DELEGATION\": \"true\"\n    }\n}"
...

Using an input file, as an argument to the configuration-values parameter, removes the need to escape and format the YAML or JSON into a flattened string for the CLI input. Using input files also allows for separation of input values from commands, which enables a better DevOps experience.

Example update add-on workflow

With the new Amazon EKS add-on API enhancements, customers can also update already created (i.e., ACTIVE) cluster add-ons and supply respective advanced configuration settings. Given a scenario where we have installed the CoreDNS add-on into our existing cluster, we now want to modify the corefile to change the cache interval to 29, and add an additional server entry.

With the following kubectl command, we can see the current corefile that CoreDNS is using.

kubectl -n kube-system get cm coredns -oyaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
...
        cache 30
...
    }
kind: ConfigMap
...

To update the add-on and change the CoreDNS corefile we’ll use the following update-addon command to pass the corefile contents via a YAML or JSON input file to the CLI command configuration-values parameter.

# Update coredns add-on corefile contents using YAML input file
aws eks update-addon --cluster-name <CLUSTER_NAME> \
--addon-name coredns --addon-version v1.8.4-eksbuild.2 \
--configuration-values 'file://example.yaml' \
--resolve-conflicts OVERWRITE

# Update coredns add-on corefile contents using JSON input file
aws eks update-addon --cluster-name <CLUSTER_NAME> \
--addon-name coredns --addon-version v1.8.4-eksbuild.2 \
--configuration-values 'file://example.json' \
--resolve-conflicts OVERWRITE

aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name coredns

{
    “addon”: {
        “addonName”: “coredns”,
        “clusterName”: “<CLUSTER_NAME>”,
        “status”: “ACTIVE”,
        “addonVersion”: “v1.8.4-eksbuild.2”,
        “health”: {
            “issues”: []
        },
…
     “  "configurationVal”es“: "{\n  ” \"corefi”e\"” \".:53 {\\n    errors\\n    health\\n rometheustes cluster.local in-addr.arpa ip6.arpa {\\n      pods insecure\\n    fallthrough in-addr.arpa ip6.arpa\\n    }\\n rometheuseus :9153\\n    forward . /etc/resolv.conf\\n    cache 29\\n    loop\\n    reload\\n    loadbalance\\n}\\nexample.org {\\n    whoami\\”}\"”n}"
    }
}

# Verify coredns corefile contents
kubectl -n kube-system get cm coredns -oyaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
...
        cache 29
...
    }
    example.org {
        whoami
    }
kind: ConfigMap
...

The preceding command updated the CoreDNS corefile, changing the cache interval to 29 and adding a server entry to handle the example.org query with the whoami plugin.

Next, we'll update the number of replicas for the coredns Deployment resource using the following CLI command.

# Update coredns add-on replica count using YAML input file
aws eks update-addon --cluster-name <CLUSTER_NAME> \
--addon-name coredns --addon-version v1.8.4-eksbuild.2 \
--configuration-values 'file://example.yaml' \
--resolve-conflicts OVERWRITE

# Update coredns add-on replica count using JSON input file
aws eks update-addon --cluster-name <CLUSTER_NAME> \
--addon-name coredns --addon-version v1.8.4-eksbuild.2 \
--configuration-values 'file://example.json' \
--resolve-conflicts OVERWRITE

{
    "update": {
...
            {
                "type": "ConfigurationValues",
                "value": "{\n    \"replicaCount\":3\n}"
            }
...

# Describe cordedns add-on
aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name coredns

{
    "addon": {
        "addonName": "coredns",
        "clusterName": "<CLUSTER_NAME>",
        "status": "ACTIVE",
...
        "configurationValues": "{\n    \"replicaCount\":3\n}"
    }
}

# Get coredns pods, verify 3 replicas
kubectl -n kube-system get po -l k8s-app=kube-dns

NAME                       READY   STATUS    RESTARTS   AGE
coredns-687f68f788-5nmv7   1/1     Running   0          19m
coredns-687f68f788-q7mvt   1/1     Running   0          19m
coredns-687f68f788-vh8xv   1/1     Running   0          5m11s

# Review coredns corefile contents, reset to defaults
kubectl -n kube-system get cm coredns -oyaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
...
        cache 30
...
    }
kind: ConfigMap

The preceding example illustrates an important aspect of updating add-ons with advanced configurations supplied in the configurationValues parameter. When the configurationValues parameter is used, advanced configurations don’t persist between add-on API calls. They are overwritten by default settings unless they are explicitly included, which is by design. If you use the configurationValues parameter and you don’t want your settings to be overwritten, then you must include them in the configurationValues parameter of each UpdateAddon operation.

In our CoreDNS UpdateAddon operation, the corefile contents were reset to default settings because we didn’t add them to the update command. To include the corefile settings with the replica settings, we would add them together in the same YAML or JSON file, as seen in the following example.

# Example YAML file with replicaCount and corefile elements
replicaCount: 3
corefile: |
    .:53 {
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 29
       loop
       reload
       loadbalance
    }
    example.org {
      whoami
    }

# Example JSON file with replicaCount and corefile elements
{
    "replicaCount":3,
    "corefile": ".:53 {\n    errors\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n    fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 29\n    loop\n    reload\n    loadbalance\n}\nexample.org {\n    whoami\n}"
}

# configurationValues field from describe-addon results
        "configurationValues": "{\n    \"replicaCount\":3,\n    \"corefile\": \".:53 {\\n    errors\\n    health\\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\\n      pods insecure\\n    fallthrough in-addr.arpa ip6.arpa\\n    }\\n    prometheus :9153\\n    forward . /etc/resolv.conf\\n    cache 29\\n    loop\\n    reload\\n    loadbalance\\n}\\nexample.org {\\n    whoami\\n}\"\n}"

It’s also important to understand when and how to use the --resolve-conflicts parameter. Only use the --resolve-conflicts OVERWRITE option when you are sure you won’t mistakenly overwrite settings in the add-on configuration. Settings that would be overwritten are those settings that were added to the add-on component configuration outside of the Amazon EKS add-on API, made via the Kubernetes API. If you know the settings that would need to be included as advanced configurations, and the settings are possible via the discovered configurationSchema, then add them to the configurationValues parameter of CreateAddon or UpdateAddon API operations.

Note: In our previous post, Amazon EKS add-ons preserve customer edits, we introduced the --resolve-conflicts PRESERVE option for the Amazon EKS API UpdateAddon operation. This option preserves edits made to add-on resources so that they aren’t overwritten during an UpdateAddon operation. The PRESERVE option can be used with the advanced configurations made through the Amazon EKS add-on API. However, we strongly recommend that edits to add-on configurations be made via the Amazon EKS add-on API.

Updating add-ons and compatibility

Amazon EKS add-ons can be updated to newer (or older) versions. If no new advanced configuration settings are supplied, the add-on API attempts to apply any existing configuration settings. If the existing configuration settings are compatible with the changed add-on version, then the update will work, preserving these existing settings.

If current add-on settings are incompatible with the target add-on version, then an error is returned by the API. For example, given an existing Amazon EKS cluster and an existing add-on with the configuration setting, {"replicas":4}, in the Kubernetes Deployment resource specification, the add-on API checks if the existing settings are compatible with the target add-on version. If the target version of the add-on now uses Kubernetes Daemonset resources instead of Deployments, the replicas setting will no longer be compatible. An attempt to update the add-on with the following command fails and returns an error from the add-on API.

aws eks update-addon --cluster-name <CLUSTER_NAME> --addon-name <ADDON_NAME> --addon-version <ADDON_VERSION>

An error occurred (InvalidParameterException) when calling the UpdateAddon operation: Configuration option "replicas" is not valid

Resetting default add-on settings via updates

For the use case where Amazon EKS add-on advanced configurations are to be removed, and default settings are desired, the add-on API UpdateAddon call is used with an empty YAML string or JSON object (empty curly braces). For example, the following AWS CLI command can be used to update an existing add-on and remove any advanced configurations. This performs an update on the add-on, resetting all advanced configurations back to their default values.

aws eks udpate-addon --cluster-name <CLUSTER_NAME> \
--addon-name <ADDON_NAME> --addon-version <ADDON_VERSION> \
--configuration-values '' \
--resolve-conflicts OVERWRITE

Conclusion

In this post, we provided an overview of newly released Amazon EKS add-ons advanced configurations support. The existing DescribeAddonVersions API call is used with the new DescribeAddonConfiguration API call to discover available advanced configurations, by examining the configurationSchema for the combination of add-on and add-on version. This JSON schema exposes configurable settings that allow customers to customize certain add-on settings. The configuration-values parameter is used to supply advanced configuration settings via YAML or JSON that follows the schema found in the configurationSchema element. If the configuration-values parameter is provided in CreateAddon and UpdateAddon calls, it overrides advanced configurations settings. If no configuration-values parameter is supplied, then the CreateAddon and UpdateAddon calls will not change those settings. The OVERWRITE option can be used to resolve potential conflicts by overwriting values in the CreateAddon or UpdateAddon operations. The PRESERVE option can be used to preserve edits from one operation to the next.

Calls to action

Give Amazon EKS add-ons a spin!

If you are running Amazon EKS and you haven’t tried add-ons yet, now is the time. The new advanced configurations add more robust configuration management to Amazon EKS add-ons. You can now discover additional settings that are underpinned by the Helm charts we use to manage add-ons in your clusters. This provides for more granular configurations, a more flexible user experience, and better fits new and existing DevOps automation.

Check out our containers roadmap!

If you have ideas about how we can improve Amazon EKS add-ons or other aspects of our container services, then please use our containers roadmap to provide us feedback and review our existing roadmap items.

Containers