AWS Partner Network (APN) Blog

How to Enable Secure Self-Service Amazon EKS Cluster Provisioning with ArgoCD, Kyverno, and Nirmata Policy Manager

By Ritesh Patel, VP of Products – Nirmata
By Marina Novikova, Sr. Partner Solutions Architect – AWS

Nirmata-AWS-Partners-2024
Nirmata
Connect with Nirmata-1

As cloud adoption accelerates, organizations spanning diverse sectors and sizes are embracing cloud-native technologies to achieve agility, adaptability, and robustness.

Containers are a leading method for packaging applications, with Kubernetes emerging as the top orchestration solution. While the rapid deployment and agility of distributed cloud-native applications offer tremendous advantages, it can also introduce complexity and potential pitfalls.

Kubernetes, though a potent and versatile tool, can be challenging to configure and operate efficiently at scale. Consequently, many enterprises leveraging Amazon Elastic Kubernetes Service (Amazon EKS) find it imperative to implement self-service cluster provisioning to navigate these complexities effectively. This enables streamlined cluster management, cost optimization, enhanced security, and adherence to compliance standards while allowing for scalability.

This approach aligns with the broader trend of DevOps and cloud-native practices, where teams aim to automate and streamline infrastructure management for faster and more efficient application development and deployment.

Cloud security at Amazon Web Services (AWS) is the highest priority. In this post, we will explore how a secure self-service cluster provisioning workflow can be enabled using AWS Controllers for Kubernetes (ACK), Kyverno, and ArgoCD.

Nirmata is the creator of Kyverno and an AWS Specialization Partner with the Containers Competency and Amazon EKS service ready designation. An AWS Marketplace Seller as well, Nirmata delivers security, compliance, and operational readiness of Kubernetes across the enterprise.

Solution Overview

For AWS Controllers for Kubernetes, Kyverno, and ArgoCD, each of these components are preconfigured with appropriate permissions to perform specific tasks.

Figure 1: Creating resource using Kyverno generate policy via ArgoCD application

Figure 1 – Creating resource using Kyverno generates policy via ArgoCD application.

AWS Controllers for Kubernetes (ACK)

AWS Controllers for Kubernetes (ACK) is an open-source project by AWS that simplifies the management of AWS resources within Kubernetes clusters. It provides custom resource definitions (CRDs) and controllers that allow you to define and operate AWS resources directly from your Kubernetes manifests.

ACK extends the Kubernetes API to support AWS resources, enabling you to use familiar Kubernetes tools and practices to manage AWS resources seamlessly.

Kyverno Policy Engine

Kyverno is an open-source Kubernetes-native policy engine created by Nirmata that runs as an admission controller and can validate, mutate, and generate any configuration data based on customizable policies.

Although other general purpose policy solutions were retrofitted to Kubernetes, Kyverno was designed ground-up for Kubernetes. Like Kubernetes, it adopts a declarative management paradigm. Kyverno policies are simply Kubernetes resources, and don’t require learning a new language. Kyverno helps secure the Kubernetes configuration by preventing misconfigurations and enhancing security.

ArgoCD

ArgoCD is an open-source continuous delivery tool designed for Kubernetes environments. It specializes in automating the deployment and management of applications on Kubernetes clusters, ensuring applications are always in the desired state, as defined in Git repositories. ArgoCD is well-suited for cloud-native and containerized applications.

Nirmata Policy Manager

Nirmata Policy Manager provides organizations with powerful policy-driven security and governance capabilities for their Kubernetes-based containerized applications.

With Nirmata Policy Manager, users can define and enforce policies that ensure compliance, security, and resource optimization within their Kubernetes clusters and applications. It allows for fine-grained control over various aspects of application deployment and management, such as access controls, resource quotas, scaling rules, and more.

Putting it All Together

Next, we’ll demonstrate how to create a management cluster on Amazon EKS. You can follow the instructions to install ACK, ArgoCD, and Kyverno on the management cluster.

Step 1: Install and Configure ACK

Install the ACK controller by following the instructions for an ACK controller. In order to use the controller, you have to configure the AWS Identity and Access Management roles for service accounts (IRSA) and give full access to the EKS service. Follow the detailed instructions to configure IRSA for the cluster.

Once the ACK controller is configured, you can verify by trying to create a cluster. Use the sample ACK YAML you can start with. Note that you’ll need to specify the cluster and node role ARNs.

Step 2: Install and Configure ArgoCD

Install ArgoCD using the instructions to install the ArgoCD Helm chart.

ArgoCD will be used to install the add-ons once the cluster has been provisioned, so it needs to be able to assume the same role that was used to create the cluster. Find detailed instructions to install and configure ArgoCD on the Amazon EKS cluster.

Once ArgoCD is installed and verified, create the appsets for the cluster add-ons:

kubectl create -f appsets/appset_kyverno.yaml -n argocd
kubectl create -f appsets/appset_kyverno_policies.yaml -n argocd

Optionally, if you want to automatically register your cluster to Nirmata Policy Manager, add the Nirmata Controller Registrator appset.

Note that you will need to edit the appset YAML file and set the API token property for your NPM account. To create the API token, go to Settings > Profile and click on Generate a new API Key. Once the API key is generated, use the Copy API Key button to copy the API key and paste it to the YAML file.

Figure 2: Configuring API token in Nirmata Policy Manager

Figure 2 – Configuring API token in Nirmata Policy Manager.

To add the appset:

kubectl create -f appsets/appset_nirmata_controller_chart.yaml -n argocd

You can verify the appsets created:

kubectl get appsets -n argocd

In ArgoCD, change the resource tracking method to annotation so it doesn’t conflict with Kyverno as described in the documentation:

kubectl edit cm -n argocd argocd-cm

apiVersion: v1
data:
  application.resourceTrackingMethod: annotation
kind: ConfigMap

Step 3: Configure Kyverno

Install Kyverno by following the instructions.

Once Kyverno is installed, update the kyverno:generate cluster role so the ArgoCD application resource can be created:

- apiGroups:
  - argoproj.io
  resources:
  - applications
  verbs:
  - create
  - update
  - patch
  - delete

Now, apply the generate policies that will be used to create the ArgoCD application that creates the EKS cluster and register the cluster with ArgoCD:

kubectl create -f policies/gmt.-cluster

The generate-application policy triggers the creation of an ArgoCD application that generates the resources to create the EKS cluster. This policy needs to be updated with the cluster role’s Amazon Resource Name (ARN), node role ARN, subnets, and any other default configuration required for cluster and node group creation.

The register-cluster policy registers newly-created ACK clusters with ArgoCD so the cluster add-ons can be deployed.

Step 4: Add CreateCluster Custom Resource Definition

Next, we need a way for users to request clusters. We will create a custom resource (CR) called CreateCluster in the management cluster; the custom resource definition can be found on GitHub. This CRD current allows a few fields such as name and desiredSize to be specified, but it can be easily extended to include other fields that can be provided in the cluster creation request.

When the CreateCluster CR is created, the generate-application policy we created above will trigger and create an ArgoCD application that creates the cluster, node group, and add-on custom resources for EKS cluster creation.

Step 5: Create an EKS Cluster

Once everything is in place, verify the setup. Apply the cluster creation request to your cluster:

apiVersion: mgmt.nirmata.io/v1alpha1
kind: CreateCluster
metadata:
  name: ack-test-cluster
spec:
  name: ack-test-cluster
  desiredSize: 2

This should initiate creation of the ArgoCD application to deploy the cluster. Within a few seconds, you should be able to go to your Amazon EKS console and view if the cluster is being created. You can also verify the cluster creation in your management cluster using the command:

kubectl get cluster -n <namespace>

The cluster creation should take around 10-15 minutes. Once it’s created (in ACTIVE status), the register-cluster policy will trigger to register the newly0created ACK cluster with ArgoCD so the cluster add-ons can be deployed. This can be verified on the ArgoCD web console or by accessing the newly-created cluster

If the Nirmata Controller Registrator appset has been configured, the cluster will show as registered in Nirmata Policy Manager.

The newly-created cluster has Kyverno and the default policies installed. You can also install other required add-ons by creating corresponding ArgoCD appsets. This cluster is ready for use by your application teams.

Conclusion

As demonstrated in this post, you can provision Amazon EKS in a secure and efficient manner using AWS Controllers for Kubernetes (ACK), ArgoCD, and Kyverno.

ACK provides developers with a familiar interface to provision cloud resources, while ArgoCD facilitates GitOps to deploy cluster add-ons from a central git repository, and Kyverno provides the automation necessary to deliver self-service cluster provisioning. Applying and enforcing policies using Kyverno also helps ensure configuration security.

Using the Nirmata Policy Manager, you get visibility into the security posture of your Kubernetes clusters without much effort and ensure the applications deployed to these clusters are compliant.

To learn more:

  • Watch the demo on how to use ACK with ArgoCD and Kyverno
  • Check the Nirmata blog to learn about Kyverno and Nirmata Policy Manager

.
Nirmata-APN-Blog-Connect-2024
.

Nirmata – AWS Partner Spotlight

Nirmata is an AWS Specialization Partner and policy-driven Kubernetes security and governance platform that enables enterprise platform security teams to deliver Kubernetes infrastructure and applications.

Contact Nirmata | Partner Overview | AWS Marketplace