Amazon EKS add-ons preserve customer edits

Introduction

As part of AWS re:Invent 2020, the Amazon Elastic Kubernetes Service (Amazon EKS) Team announced the release of Amazon EKS add-ons. The addition of add-ons was driven by customer feedback and a desire to simplify the management of commonly used operational software. With add-ons, you can configure, deploy, and update operational software, which provides key functionality to support your Kubernetes applications. Add-ons include critical tools for cluster networking such as the Amazon VPC CNI, as well as operational software for observability, storage, management, and security. By using add-ons, you can configure Amazon EKS to start the add-on software on the cluster and deploy new versions of the add-on with a single command.

Every add-on includes the latest security patches and bug fixes and is validated by AWS to work with Amazon EKS. This reduces the amount of work you need to do in order to start, manage, and upgrade production-ready Kubernetes clusters, which helps to keep your clusters stable and secure.

Currently, Amazon EKS add-ons support the following software:

Server-side apply and overwriting edits

Amazon EKS add-ons uses a feature, made available in Kubernetes version 1.18, called server-side apply. Server-side apply migrates the functionality of kubectl apply to the Kubernetes application programming interface (API) server and provides fine-grained control over resolving conflicts, which can happen during configuration changes. Server-side apply introduces a new configuration merging algorithm and field ownership tracking within the Kubernetes control plane. This allows Kubernetes to detect and resolve conflicts automatically.

Once add-ons are added to an Amazon EKS cluster, you can modify the configuration of the respective managed software through the Kubernetes API. Until now, however, managed add-ons prevented configuration drift by automatically overwriting configurations every 15 minutes. This meant that any changes to managed add-ons, made through the Kubernetes API after add-on creation, were overwritten by the automated drift-prevention process. While preventing drift is a desirable goal, overwriting subsequent edits to software managed by add-ons also prevents customers from making deliberate and needed changes to Kubernetes resources under add-on management.

Solution overview

Amazon EKS add-ons behavior update: Preserving customer edits

With the latest release of Amazon EKS add-ons, we have added functionality to preserve customer edits to software managed by add-ons. Instead of overwriting software configurations every 15 minutes, we perform health checks of software managed by add-ons. This means that add-ons no longer write or overwrite anything on the cluster during steady-state operations. The only time that add-ons now write to a cluster is during create, update, and delete add-on operations that are triggered by customers.

In addition to no longer overwriting configurations every 15 minutes, a new ResolveConflicts option, PRESERVE, has been added to the Amazon EKS API. This new PRESERVE option preserves any in-cluster configuration that was made through the Kubernetes API to add-on software, which is outside of the scope of add-ons. As seen below, in the Example add-on update workflow, the --resolve-conflicts PRESERVE parameter and argument can be used with the aws eks update-addon AWS Command Line Interface (AWS CLI) command to prevent the add-on update operation from overwriting any edits made to the add-on software, after initial creation.

Example add-on update workflow

Step 1 – Create the add-on in the cluster:

aws eks create-addon --cluster-name <CLUSTER_NAME> --addon-name coredns --addon-version v1.8.4-eksbuild.1 --resolve-conflicts OVERWRITE

Note: To avoid lost configuration settings, you must wait to edit the add-on until after the add-on creation process has finished. The following AWS CLI command can be used to check the status of an add-on:

aws eks describe-addon —cluster-name <CLUSTER_NAME> —addon-name <ADDON_NAME> | grep status
"status": "CREATING",

Step 2 – Modify desired configuration for the add-on in the cluster through the Kubernetes API:

kubectl -n kube-system edit configmap coredns

...
Corefile: |
...
  cache 15 // updated cache value
...

Note: To view the server-side apply managed fields for a Kubernetes resource, you can use the --show-managed-fields parameter, as seen below:

kubectl -n kube-system get configmap coredns -o yaml --show-managed-fields

Step 3 – When performing an add-on update, provide the --resolve-conflicts PRESERVE parameter and argument:

aws eks update-addon --cluster-name <CLUSTER_NAME> --addon-name coredns --addon-version v1.8.5-eksbuild.1 --resolve-conflicts PRESERVE

Step 4 – Confirm that the update was successful and that the edited configuration was preserved:

aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name coredns
{
  "addon": {
    "addonName": "coredns",
    "clusterName": <CLUSTER_NAME>,
    "status": "ACTIVE",
    "addonVersion": "v1.8.5-eksbuild.1",
    "health": {
      "issues": []
    },...

kubectl -n kube-system get deployment coredns -o yaml

...
  image: <IMAGE_REGISTRY>/eks/coredns:v1.8.5-eksbuild.1
... 

kubectl -n kube-system get configmap coredns -o yaml

...
Corefile: |
...
  cache 15 // updated cache value
...

When performing an add-on update operation, providing the PRESERVE option causes the operation to compare what currently exists on the cluster with what the add-on expects, to identify any field that differs or has been modified externally. Once identified, when performing the related add-on update operation, the fields that have been modified remain untouched while everything else is updated. This allows configurations made directly through the Kubernetes API to be preserved when updating an add-on. Default create and update behavior (failing on conflict during an operation) remains unchanged.

Note: Amazon EKS add-ons will allow any modified field to be preserved during an update operation. This means that when you use the PRESERVE option, it is possible to modify a field that becomes preserved, while simultaneously incompatible in a different version of the add-on. Therefore, it is recommended to follow best practices, and test any updates on non-production clusters, with desired and specific configuration changes, to confirm that the updates work as intended before introducing like changes to production clusters.

Preventing conflicts in create and update operations

From the beginning, the Amazon EKS API has supported detecting, preventing, or overwriting conflicts during add-on create and update operations. This functionality remains unchanged. If you attempt to create or update a managed add-on after editing a field that is or will be managed by the respective add-on and you do not provide the OVERWRITE option, existing conflicts cause the operation to fail, as seen in the following code:

kubectl -n kube-system edit configmap kube-proxy-config

...
    iptables:
...
      syncPeriod: 40s
...

aws eks create-addon --cluster-name <CLUSTER_NAME> --addon-name kube-proxy

aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name kube-proxy

{
    "addon": {
        "addonName": "kube-proxy",
        "clusterName": "<CLUSTER_NAME",
        "status": "CREATE_FAILED",
        "addonVersion": "v1.22.6-eksbuild.1",
        "health": {
            "issues": [
                {
                    "code": "ConfigurationConflict",
                    "message": "Conflicts found when trying to apply. Will not continue due to resolve conflicts mode. Conflicts:\nConfigMap kube-proxy-config - .data.config"
                }
            ]
        },
        "addonArn": "arn:aws:eks:<REGION>:<AWS_ACCOUNT>:addon/addons/kube-proxy/26c11eb4-819b-f299-7909-74c887051e48",
        "createdAt": "2022-07-26T22:07:49.300000-04:00",
        "modifiedAt": "2022-07-26T22:08:01.229000-04:00",
        "tags": {}
    }
}

To get around potential conflicts, you can use the OVERWRITE option with create operations, or the OVERWRITE or PRESERVE options with update operations, as seen below, in the aws eks create-addon command:

aws eks create-addon --cluster-name <CLUSTER_NAME> --addon-name kube-proxy --resolve-conflicts OVERWRITE

The previous command can be used even if the previous operation failed due to a detected conflict:

aws eks describe-addon --cluster-name <CLUSTER_NAME> --addon-name kube-proxy | grep status
        "status": "ACTIVE",

After add-on creation, you can edit the add-on resources through the Kubernetes API, and then use the PRESERVE option with an add-on update operation to update the add-on while preserving your edits.

Another use case for the OVERWRITE option is to reset Kubernetes resource configurations. If you modify the Kubernetes resources of software that is, or can be, managed by add-ons, you can use the OVERWRITE option of the create or update operations to reset the configuration back to a known good configuration, prescribed by the add-on.

Preserving Kubernetes resources beyond add-on deletion

If you no longer want or need an Amazon EKS add-on in your cluster, you can delete the add-on and choose whether or not to retain the underlying Kubernetes resources. As seen below, using the --preserve flag with the delete-addon command will retain the underlying Kubernetes resources, while removing the add-on management from the software:

aws eks delete-addon --cluster-name <CLUSTER_NAME> --addon-name <ADDON_NAME> --preserve

Note: When modifying production configurations, it is a best practice to ensure that configurations can be reapplied through solutions like back-up and restore, or GitOps, should the need arise.

Conclusion

In this post, we showed you how Amazon EKS add-ons reduce the heavy-lifting required to manage common operational software in your clusters. As mentioned in the previous sections of this post, every add-on includes the latest security patches and bug fixes, and is validated by AWS to work with Amazon EKS. Add-ons are an evolution of Amazon EKS, which began with us managing the control plane (managed upgrades) and data plane (managed node groups) and progressed with managing the common operational cluster software (Amazon EKS add-ons). With the latest PRESERVE option for the Amazon EKS API add-on update operations, you can modify configurations of software under add-on management without losing your specific edits. You can use add-ons to help manage software, while still customizing the software for your individual use cases.

Calls to action

Give Amazon EKS add-ons a spin!

If you are running Amazon EKS and you haven’t tried add-ons yet, now is the time. Maybe you’ve held off due to the aforementioned drift protection and didn’t want your Kubernetes resource edits overwritten. With the new add-on update --resolve-conflicts PRESERVE option as well as the existing add-on delete --preserve option, you can use add-ons while preserving your edits. For the future, efforts are underway to develop a more robust configuration management system for Amazon EKS add-ons, so please stay tuned for future announcements!

Check out our containers roadmap!

If you have ideas about how we can improve Amazon EKS add-ons or other aspects of our container services, please use our containers roadmap to provide us feedback and review our existing roadmap items.

Containers