Using sidecar injection on Amazon EKS with AWS App Mesh

NOTICE: October 04, 2024 – This post no longer reflects the best guidance for configuring a service mesh with Amazon EKS and its examples no longer work as shown. Please refer to newer content on Amazon VPC Lattice.

——–

AWS App Mesh works on the sidecar pattern where you must add containers to extend the behavior of existing containers. Kubernetes offers mutating admission controllers that allow operations teams to automate sidecar injection. In this post, I discuss the basics of the sidecar pattern and Kubernetes admission controllers and demonstrate how the App Mesh Sidecar Injector works on Amazon Elastic Kubernetes Service (Amazon EKS) or your Kubernetes installation on Amazon EC2.

Sidecar is a popular microservices pattern.

What are sidecars?

Sidecar is a microservices design pattern where a companion service runs next to your primary microservice, augmenting its abilities or intercepting resources it is utilizing. In the case of App Mesh, a sidecar container, Envoy, is used as a proxy for all ingress and egress traffic to the primary microservice. Using this sidecar pattern with Envoy we create the backbone of the service mesh, without impacting our applications.

Injectors

Kubernetes can hook into actions on Kubernetes objects before the system executes them. This allows two main functions: validation and mutation. Validation works as a gatekeeper for any operation on any resource in Kubernetes, allowing you to block actions before they execute. Mutation allows you to change the specification of a Kubernetes resource before initiation. Mutation provides the functionality for App Mesh injector.

Solution Walkthrough

This example deploys the App Mesh sidecar injector, then tests to see if injection works.
This process involves:

Deploying the App Mesh Sidecar Injector
- Verifying the sidecar injector deployed correctly
Testing the sidecar injector functionality
- Creating a namespace with sidecar injection enabled
- Deploying a pod to the namespace
- Verifying the sidecar injected successfully

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account
A running EKS cluster
A service mesh configured in AWS App Mesh
Working knowledge of Kubernetes basics

Sidecar injection

To learn more about the sidecar injector, see the aws-app-mesh-inject GitHub repo.

Step 1: Deploy the sidecar injector

We deploy the App Mesh Sidecar Injector to the Kubernetes cluster using preconfigured bash scripts.

After your cluster is up and running, export your mesh name:
```
export MESH_NAME=mynewmesh
```

Install the sidecar injector in the namespace “appmesh-inject” :

curl https://raw.githubusercontent.com/aws/aws-app-mesh-inject/master/scripts/install.sh | bash

Verify the installation:

kubectl get pods -n appmesh-inject
NAME                         READY STATUS  RESTARTS AGE
aws-app-mesh-inject-XXXXXXXX 1/1   Running 0        21s

Step 2: Testing sidecar injector

To see if the injector works, deploy a pod in a namespace labeled to enable sidecar injection.

Create a namespace with sidecar injection enabled:
```
cat <<EOF  | kubectl apply -f –
apiVersion: v1
kind: Namespace
metadata:
  name: appmesh-inject-test
  labels:
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
EOF
```
Labeling the namespace appmesh.k8s.aws/sidecarInjectorWebhook: enabled enables sidecar injection on pods in that namespace.

Deploy a pod and see if the injector works correctly.

cat <<EOF  | kubectl apply -f –
apiVersion: v1
kind: Pod
metadata:
  namespace: appmesh-inject-test
  name: nginx
  labels:
    name: nginx
  annotations:
    appmesh.k8s.aws/virtualNode: "test"
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
EOF

You can see in the pod itself if the sidecar successfully intercepted and mutated it.
```
kubectl get pods -n appmesh-inject-test
NAME  READY STATUS  RESTARTS AGE
nginx 2/2   Running 0        17s
```
The notation 2/2 indicates your NGINX pod and the sidecar running next to it.

Sidecar configuration

App Mesh assigns the pod to the virtual node named <namespace>-<deployment-name>. In this example, it would be appmesh-inject-test-nginx.
Using this Virtual Node name, you can configure routing to your pod in App Mesh.
In addition to sidecar injection, you can also override some default values with labels in your pod specification.
You can specify the mesh the pod is part of by setting the mesh name in the pod annotation:

appmesh.k8s.aws/mesh: my-mesh

Normally, Envoy handles all ports in the pod, but you can override by using the ports annotation:

appmesh.k8s.aws/ports: 8079,8080

You can override the virtual node assigned to the pod by setting the virtualNode annotation:

appmesh.k8s.aws/virtualNode: my-app

You can also disable sidecar injector for a specific pod:

appmesh.k8s.aws/sidecarInjectorWebhook: disabled

In cases where a proxy can interfere with the functionality of egress traffic you can ignore specific ports for egress:

appmesh.k8s.aws/egressIgnoredPorts: "22"

Cleaning up

To remove the sidecar injector from your cluster, run the following commands:

kubectl delete namespace appmesh-inject
kubectl delete mutatingwebhookconfiguration aws-app-mesh-inject
kubectl delete clusterrolebindings aws-app-mesh-inject-binding
kubectl delete clusterrole aws-app-mesh-inject-cr

Conclusion

There are several widely accepted patterns you can adopt in designing your microservice architecture. The Adapter Pattern can be used to mutate disparate container outputs into one unified format, or the Ambassador Pattern, which can be used to negotiate the connection to external resources such as distributing read and write traffic to a clustered data store. The Sidecar Pattern is a popular pattern because it allows you to compose pods from containers with independent but complimentary functionality. This allows for code reuse and teams to work independently. Sidecar injection does not have to be a burden on your developers and operations by using mutating webhooks and the App Mesh Sidecar Injector to easily automate the process for injection and configuration.

You can learn more about App Mesh in our documentation or to contribute to the App Mesh Sidecar Injector please check out our GitHub repo. Please leave any comments below or reach out to me via Twitter!

Containers