Using HashiCorp Consul Service Mesh with AWS Lambda

By Rosemary Wang, Developer Advocate – HashiCorp
By Young Jeong, Partner Solutions Architect – AWS
By Welly Siauw, Principal Partner Solutions Architect – AWS

HashiCorp

As companies grow, they often find themselves needing to migrate applications to new architectures that better fit their needs. For example, you may want to re-platform from Amazon Elastic Container Service (Amazon ECS) to AWS Lambda to benefit from millisecond cost metering.

During your application migration, you need to gradually replace services from one architecture to another with a temporary façade and maintain overall system functionality.

The façade can help shape traffic across platforms by controlling migration of traffic to services on new platforms. Separating the concern of traffic management to an external façade such as HashiCorp Consul reduces refactoring effort within service code.

Consul is a service networking solution to automate network configurations, discover services, and enable secure connectivity across any cloud or runtime. Its service mesh capabilities can be run across multiple compute platforms, including AWS Lambda, Amazon ECS, Amazon Elastic Compute Cloud (Amazon EC2), and Amazon Elastic Kubernetes Service (Amazon EKS).

By using Consul as a façade, you can control traffic to and from services across different platforms from a single interface. Consul integrates with AWS Lambda by treating functions as part of its service mesh. As you refactor services to functions, you can set up Consul to manage traffic to and from functions to maintain overall system functionality during testing and migration.

In this post, we will introduce a solution for routing service request from Amazon ECS to Lambda using HashiCorp Consul. HashiCorp is an AWS Competency Partner and Lambda Service Ready product that provides consistent workflows to provision, secure, connect, and run any infrastructure for any application.

Solution Overview

This solution uses a simple microservice consisting of a Greeter service that calls the Greeting service for a list of greetings in different languages, and the Name service for random names.

Consider a scenario where you refactor the Greeter service to run as a Lambda function. After migrating the Greeter service to Lambda, you can gradually cutover traffic from the original service on Amazon ECS to Lambda function running the new service.

Figure 1 – Greeter service architecture with Amazon ECS and AWS Lambda.

We will use HashiCorp Terraform to deploy an ECS cluster, the sample microservices running on ECS, the Greeter Lambda function, Consul server, and the supporting services.

To allow communication to and from the Lambda function into the service mesh, we use three components of Consul:

First, we use Consul Lambda Registrator to automatically register and de-register Lambda functions with Consul based on the function’s tags.
Second, we use Consul Terminating Gateway to allow connectivity from services running on ECS to external destination such as a Lambda function.
Third, we use Consul Mesh Gateway to allow connectivity from Lambda back to the service mesh.

The Greeter Lambda function uses the Consul Lambda extension layer to send requests to services in the service mesh via Consul Mesh Gateway. Consul uses intentions to allow services to communicate to each other.

Both the Consul Terminating Gateway and Consul Mesh Gateway ensure that Consul intentions are enforced for communication between services in the service mesh.

Figure 2 – Greeter service architecture with ECS, Lambda, and HashiCorp Consul.

Prerequisites

To launch this solution, you need to install the following prerequisites:

AWS Command Line Interface (AWS CLI) with an AWS Identity and Access Management (IAM) credential configured from your AWS account.
HashiCorp Terraform CLI for deploying infrastructure resources.
HashiCorp Consul CLI for inspecting Consul configuration.
Docker for building and managing container image
Git and jq installed.

This walkthrough assumes basic understanding of Terraform operation and syntax. You can learn more about Terraform from HashiCorp’s tutorials.

Configure ECS, Container Images, and Consul Infrastructure

To start deploying the solution, clone the example code, which includes the Terraform configuration for the solution.

git clone https://github.com/aws-samples/amazon-ecs-lambda-consul-example
cd amazon-ecs-lambda-consul-example/terraform

Create a file named secrets.auto.tfvars. This file should contain three variables—the name of an EC2 keypair (for accessing the bastion), a list of allowed IP addresses for load balancers and the bastion, and AWS region to deploy the sample solution.

Replace the placeholder below with your own values. We recommend limiting the IP CIDR block to your own public IP address.

cat << EOF > secrets.auto.tfvars
ec2_key_pair_name = "<your EC2 keypair>"
ingress_cidrs = [“<list of allowed IP addresses”]
region = “<your AWS region>”
EOF

Run terraform init to initialize the environment and terraform apply to create the infrastructure.

terraform init
terraform apply

Use terraform output to retrieve the Consul load balancer domain name system (DNS). The output includes other attributes such as the ingress load balancer DNS and Amazon Elastic Container Registry (Amazon ECR) repository names for scripts to use.

$ terraform output

Before deploying the services to ECS, run the build.sh script in the services/directory to build and push the required container images to ECR repositories.

cd .. && bash services/build.sh

Set the `CONSUL_HTTP_ADDR` environment variable so you can access Consul via CLI. You can also use the Consul load balancer value to view the Consul user interface (UI) from browser.

cd terraform
export CONSUL_HTTP_ADDR=$(terraform output -raw consul_server_lb_address)

Consul registered the microservices, terminating the gateway and mesh gateway in its service catalog. Run the command below to review the list of services in Consul.

consul catalog services

Deploy Amazon ECS Services

We start by deploying the original microservices in the Amazon ECS cluster. We use the hashicorp/consul-ecs/aws/mesh-task Terraform module that adds additional sidecars to join the task to service mesh.

Figure 3 – Amazon ECS service architecture with Consul.

To deploy the ECS services, open the toggles.auto.tfvars file from the Terraform directory. Set the value deploy_ecs_services = true and then save the file.

deploy_ecs_services  = true
deploy_consul_lambda = false
deploy_lambda        = false
migrate_to_lambda    = false

The deploy_ecs_services toggle triggers a local Terraform module in services.tf to deploy the microservices. Run the command below to apply the configuration.

terraform apply

The Ingress service provides access to the Greeter service, which communicates with the Greeting and Name services and returns a full greeting. Run the command below to list all services in ECS.

aws ecs list-services --cluster greetings

Consul registers each of the ECS services and their service mesh proxies into its service catalog. Run the command below to list all services in Consul. Note the addition of the Consul side car proxy that was automatically added to each of services.

consul catalog services

Consul uses intentions to allow services to communicate to each other. We define the intentions in the terraform/services/consul.tf file. Name and Greeting services allows traffic from Greeter, while Greeter allows traffic from Ingress.

Try to access the Ingress service using the command below.

curl $(terraform output -raw ingress_lb_address)

Next, you will migrate the Greeter service to a Lambda function. Using Consul service mesh, you can split traffic routing from the Ingress service on ECS to the Greeter Lambda function.

Configure Consul Integration with AWS Lambda

Consul integrates with AWS Lambda so you can invoke functions from the service mesh and call mesh services from functions. First, each Consul client must have the serverless plugin enabled in its agent configuration. Below is the example of Consul agent configuration.

connect {
 enabled = true,
 enable_serverless_plugin = true
}

Second, the Consul Lambda registrator must be deployed in the AWS account and region where your Lambda function is running. Consul Lambda registrator synchronizes Lambda functions to Consul’s service catalog based on events from AWS CloudTrail.

The Consul Lambda registrator function runs in response to the creation of a new Lambda function. Based on the function’s tags, the registrator will register the function into Consul’s service catalog.

Figure 4 – Consul service catalog registration workflow for Lambda.

To deploy Consul Lambda registrator, open the file toggles.auto.tfvars, set the value deploy_consul_lambda = true and save the file.

deploy_ecs_services  = true
deploy_consul_lambda = true
deploy_lambda        = false
migrate_to_lambda    = false

The deploy_consul_lambda toggle enables the consul-lambda-registrator module inside the consul-lambda.tf file. The configuration deploys the Consul Lambda registrator and sets the frequency of synchronization of registrations from Lambda to Consul every one minute.

Run the command below to apply the configuration.

terraform apply

After deploying the Consul Lambda registrator function, you can now refactor the Greeter service to a Lambda function and register it with Consul’s service mesh.

Deploying Greeter Lambda Function

The Greeter Lambda function must be built and packaged before you can deploy it. Run the command below to start the build process.

cd .. && bash lambda/build.sh

The terraform/lambda/greeter.tf file creates the Greeter function with three Consul-specific tags. Consul Lambda registrator uses these tags to register the service in the mesh:

Tag ”serverless.consul.hashicorp.com/v1alpha1/lambda/enabled” = “true” enables Consul service registration.
Tag ”serverless.consul.hashicorp.com/alpha/lambda/payload-passthrough” = “true” enables passing the plaintext greeting payload instead of JSON.
Tag ”serverless.consul.hashicorp.com/alpha/lambda/invocation-mode” = “ASYNCHRONOUS” indicates the function will be invoked asynchronously.

To trigger Terraform to deploy Greeter Lambda function, open the file toggles.auto.tfvars and set deploy_lambda = true.

deploy_ecs_services  = true
deploy_consul_lambda = true
deploy_lambda        = true
migrate_to_lambda    = false

Deploy the Greeter Lambda function using Terraform from your terminal.

cd terraform
terraform apply

The Consul Lambda registrator uses the tags on the Greeter function to register it to Consul. Run the command below to confirm that Consul has a “greeter-lambda” service in its catalog.

consul catalog services | grep greeter-lambda

Invoke Lambda from ECS Services

The Ingress service invokes the Greeter Lambda function using a Consul Terminating Gateway. The gateway requires sufficient IAM permissions to invoke the Greeter function.

The terraform/lambda/iam.tf file creates a policy with permissions to lambda:InvokeFunction and attaches it to the terminating gateway’s IAM role. Below is the snippet of Terraform IAM policy used by Consul Terminating Gateway.

resource "aws_iam_policy" "invoke_lambda" {
 name   = "ecs-invoke-lambda"
 path   = "/${var.name}/"
 policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "lambda:InvokeFunction"
     ],
     "Resource": "${aws_lambda_function.greeter.arn}"
   }
 ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "terminating_gateway_lambda" {
 role       = var.terminating_gateway_role_name
 policy_arn = aws_iam_policy.invoke_lambda.arn
}

To manage traffic from a service in the mesh to Lambda, you register the function to the Consul Terminating Gateway. The terraform/consul-terminating-gateway.tf creates a configuration entry in Consul to add the greeter function to the gateway.

resource "consul_config_entry" "terminating_gateway" {
 count = var.deploy_lambda ? 1 : 0

 depends_on = [
   module.dev_consul_server,
   aws_instance.consul_terminating_gateway
 ]

 name = local.terminating_gateway_service_name
 kind = "terminating-gateway"

 config_json = jsonencode({
   Services = [
     {
       Name = "greeter-lambda"
     }
   ]
 })
}

The Consul terminating gateway should have the Greeter function as a linked service. This allows services in Consul service mesh to invoke the Lambda function.

Run the command below to examine the terminating gateway in Consul and determine if it has a linked service for the Greeter Lambda function.

curl -s $CONSUL_HTTP_ADDR/v1/catalog/service/terminating-gateway | \
 jq '.[0].ServiceTaggedAddresses'

Call ECS Services from Lambda

Consul mesh gateway enables service mesh traffic to be routed between different Consul data centers. A data center is the smallest unit of Consul infrastructure that can perform basic Consul operations.

In this solution, the mesh gateway treats the ECS cluster and Lambda function as individual data centers. The Lambda function calls the Consul mesh gateway’s address and port to access other Consul services running on ECS.

This solution uses the hashicorp/consul-ecs/aws/gateway-task to create a mesh gateway as an ECS service running in AWS Fargate. You can inspect its attributes in the terraform/consul-mesh-gateway.tf file. For each Lambda function that calls a service in Consul’s service mesh, the Consul Lambda extension is added as a layer to the function.

resource "aws_lambda_layer_version" "consul_lambda_extension" {
 layer_name       = "consul-lambda-extension"
 filename         = var.consul_lambda_extension_file_path
 source_code_hash = filebase64sha256(var.consul_lambda_extension_file_path)
 description      = "Consul service mesh extension for AWS Lambda"
}

Several Lambda environment variables are also added for the Consul mesh gateway address, port, and the upstream services the function calls.

The CONSUL_SERVICE_UPSTREAMS environment variable includes a list of service and local port mappings for the extension. This allows the Greeter function to communicate over “localhost” to the corresponding local ports for the Name and Greeter services.

A snippet of the Greeter function environment variables can be found below.

resource "aws_lambda_function" "greeter" {
 function_name = "greeter-lambda"

 # omitted for clarity

 layers = [aws_lambda_layer_version.consul_lambda_extension.arn]

 environment {
   variables = {
     NAME_URL     = "http://localhost:${var.name_port}"
     GREETING_URL = "http://localhost:${var.greeting_port}"

     CONSUL_EXTENSION_DATA_PREFIX = "/${var.name}"
     CONSUL_MESH_GATEWAY_URI      = var.mesh_gateway_uri
     CONSUL_SERVICE_UPSTREAMS     = "name:${var.name_port},greeting:${var.greeting_port}"
   }
 }
}

With the Greeter Lambda connected to the service mesh, you can start the migration from the Greeter ECS service to the Lambda function.

Migrate from ECS to Lambda

A Consul service splitter splits the incoming traffic request between the Greeter ECS service and Lambda function. The service splitter sends half of all requests to the Greeter ECS service and the other half to the Lambda function.

Below is the snippet of Consul configuration for service splitter from the migrate.tf file.

resource "consul_config_entry" "lambda_redirect" {
 count = var.migrate_to_lambda ? 1 : 0
 name  = "greeter"
 kind  = "service-splitter"

 config_json = jsonencode({
   Splits = [
     {
       Weight  = 50
       Service = "greeter"
     },
     {
       Weight  = 50
       Service = "greeter-lambda"
     },
   ]
 })
}

To begin the migration, update the terraform/toggles.auto.tfvars file and set migrate_to_lambda = true.

deploy_ecs_services  = true
deploy_consul_lambda = true
deploy_lambda        = true
migrate_to_lambda    = true

Save the file and run the command below to deploy the service splitter.

terraform apply

Validating the Migration

To test the traffic split, generate a few requests from the Ingress load balancer. Half of the responses should result in a greeting from the ECS service while the others from Lambda.

Run the curl command below several times to see different output from ECS and Lambda.

curl $(terraform output -raw ingress_lb_address)

From ip-10-0-6-79.ec2.internal: From ip-10-0-3-61.ec2.internal: Hi (ip-10-0-2-33.ec2.internal) Jennifer (ip-10-0-2-133.ec2.internal)

curl $(terraform output -raw ingress_lb_address)

From ip-10-0-6-79.ec2.internal: "From lambda: Hi (ip-10-0-2-33.ec2.internal) Mary (ip-10-0-2-133.ec2.internal)"

As you test the function over time, you can increase the weight of the Consul service splitter to send more traffic to the Lambda function. To do that, open the file terraform/migrate.tf and change the weight value for “greeter” and “greeter-lambda” accordingly.

Cleanup

To avoid incurring charges on AWS resources deployed as part of this solution, toggle off each component inside the toggles.auto.tfvars files and run terraform apply. We recommend this approach to avoid any race-condition before running terraform destroy command.

Conclusion

In this post, we showed you how Consul’s integration with AWS Lambda can help migrate a service from Amazon ECS to Lambda.

We showed how ECS service invokes the function and the function can call ECS services part of Consul service mesh. By treating the function as part of Consul service mesh, you can gradually manage traffic to new functions and across compute platforms.

Some key components from this solution are:

Use Consul Lambda registrator to register Lambda functions to Consul.
Consul Terminating Gateway allows Consul services to invoke Lambda functions.
Consul mesh gateway allows Lambda functions to call Consul services.

You can find the full sample code in this GitHub repository. To learn more about HashiCorp Consul, visit the HashiCorp Learn portal and sign up for a free trial of Consul on HashiCorp Cloud Platform. Additionally, you can check out HashiCorp solutions in AWS Marketplace.

.

.

HashiCorp – AWS Partner Spotlight

HashiCorp is an AWS Competency Partner that provides consistent workflows to provision, secure, connect, and run any infrastructure for any application.

Contact HashiCorp | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog