Fine-tuning blue/green deployments on application load balancer

In a traditional approach to application deployment, you typically fix a failed deployment by redeploying an older, stable version of the application. Redeployment in traditional data centers is typically done on the same set of resources due to the cost and effort of provisioning additional resources. Applying the principles of agility, scalability, and automation capabilities of AWS can shift the paradigm of application deployment. This enables a better deployment technique called blue/green deployment.

Blue/green deployments provide near-zero downtime release and rollback capabilities. The fundamental idea behind blue/green deployment is to shift traffic between two identical environments that are running different versions of your application. The blue environment represents the current application version serving production traffic. In parallel, the green environment is staged running a newer version of your application. After the green environment is ready and tested, production traffic is redirected from blue to green. If any problems are identified, you can roll back by reverting traffic to the blue environment.

Canary deployments are a pattern for the slow rollout of new version of an existing application. The canary deployments incrementally deploy the new version, making it visible to new users in a slow fashion. As you gain confidence in the deployment, you can deploy it to replace the current version in its entirety.

AWS provides several options to help you automate and streamline your blue/green deployments and canary deployments, one such approach is using Application Load Balancer weighted target group feature. In this post, we will cover the concepts of target group stickiness, load balancer stickiness, connection draining and how they influence traffic shifting for canary and blue/green deployments when using the Application Load Balancer weighted target group feature .

Application Load Balancer weighted target groups

A target group is used to route requests to one or more registered targets like Amazon Elastic Compute Cloud (Amazon EC2) instances, fixed IP addresses, or AWS Lambda functions, among others. When creating a load balancer, you create one or more listeners and configure listener rules to direct the traffic to a target group.

Application Load Balancers now support weighted target groups routing. With this feature, you can add more than one target group to the forward action of a listener rule, and specify a weight for each group. For example, when you define a rule as having two target groups with weights of 9 and 1, the load balancer routes 90% of the traffic to the first target group and 10% to the other target group. You can create and configure your weighted target groups by using AWS Console , AWS CLI or AWS SDK.

For more information, see How do I set up weighted target groups for my Application Load Balancer?

Target group level stickiness

You can set target group stickiness to make sure clients get served from a specific target group for a configurable duration of time to ensure consistent experience. Target group stickiness is different from the already existing load balancer stickiness (also known as sticky sessions). Sticky sessions make sure that the requests from a client are always sticking to a particular target within a target group. Target group stickiness only ensures the requests are sent to a particular target group.

You can enable target group level stickiness using the AWS Command Line Interface (AWS CLI) with the TargetGroupStickinessConfig parameter, as shown in the following CLI command:

aws elbv2 modify-listener \
    --listener-arn " < LISTENER ARN > " \
    --default-actions \
    '[{
       "Type": "forward",
       "Order": 1,
       "ForwardConfig": {
          "TargetGroups": [
             {"TargetGroupArn": "<Blue Target Group ARN>", "Weight": 90}, \
             {"TargetGroupArn": "<Green Target Group ARN>", "Weight": 10}, \
          ],
          "TargetGroupStickinessConfig": {
             "Enabled": true,
             "DurationSeconds": 120
          }
       }
    }]'

In the next sections, we will see how to fine-tune weighted target group configuration to achieve effective canary deployments and blue/green deployments.

Canary deployments with Application Load Balancer weighted target group

The canary deployment pattern allows you to roll out a new version of your application to a subset of users before making it widely available. This can be helpful in validating the stability of a new version of the application or performing A/B testing.

For this use case, you want to perform canary deployment for your application and test it by driving only 10% of the incoming traffic to your new version for 12 hours. You need to create two weighted target groups for your Application Load Balancer and use target group stickiness set to a duration of 12 hours. When target group stickiness is enabled, the requests from a client are sent to the same target group for the specified time duration.

Figure 1: Blue and green target groups with weights 90 and 10 for canary deployment

We can define a rule as having two target groups, blue and green, with weights of 90 and 10, respectively, and enable target group level stickiness with a duration of 12 hours (43,200 seconds). The following table summarizes this configuration. See the following CLI command:

aws elbv2 modify-listener \
    --listener-arn " < LISTENER ARN > " \
    --default-actions \
    '[{
       "Type": "forward",
       "Order": 1,
       "ForwardConfig": {
          "TargetGroups": [
             {"TargetGroupArn": "<Blue Target Group ARN>", "Weight": 90}, \
             {"TargetGroupArn": "<Green Target Group ARN>", "Weight": 10}, \
          ],
          "TargetGroupStickinessConfig": {
             "Enabled": true,
             "DurationSeconds": 43200
          }
       }
    }]'

At this point, the users with existing sessions continue to be sent to the blue target group running version 1, and 10% of the new users without a session are sent to the green target group up to 12 hours running version 2, as illustrated in the following diagram.

Figure 2: Blue-green deployment architecture with 90% blue traffic and 10% green traffic.

When you’re confident that the new version is performing well and stable, you can update the target group weights for your blue and green target groups to be 0% and 100%, respectively, to ensure that all the traffic is shifted to your green target group. You may still see some traffic flowing into the blue target group for existing users with active session whose target group stickiness duration (in this case target group stickiness duration is 12 hours) has not expired.

Recommendation: As illustrated above, target group stickiness duration still influences the traffic shift between blue and green targets. So we recommend you to reduce the target group stickiness duration from 12 hours to 5 minutes or less depending upon your use case to ensure that the existing users going to the blue target group also fully transition to the green target group at the earliest. Some of our customers are using target group stickiness duration as 5 minutes to shift their traffic to green target group after successful canary testing.

The recommended value of stickiness may vary across application types. For example, for a typical 3-tier front-end deployment, lower target group stickiness value is desirable. However, for middle tier deployment, the target group stickiness duration value may need to be higher to account for longer transactions.

Blue/green deployments with Application Load Balancer weighted target group

For this use case, you want you perform blue/green deployment for your application to provide near-zero downtime release and rollback capabilities. You can create two weighted target groups called blue and green with the following weights applied as an initial configuration.

Figure 3: Blue/green deployment configuration with blue target group 100% and green target group 0%

When you’re ready to perform the deployment, you can change the weights for blue and green targets groups to be 0% and 100%, respectively, to shift the traffic completely to your newer version of the application.

Figure 4: Blue/green deployment configuration with blue target group 0% and green target group 100%

When you’re performing blue/green deployment using weighted target groups, the recommendation is to not enable target group level stickiness so that traffic shifts immediately from the blue target group to the green target group. See the following CLI command:

aws elbv2 modify-listener \
    --listener-arn "<LISTENER ARN>" \
    --default-actions \
    '[{
       "Type": "forward",
       "Order": 1,
       "ForwardConfig": {
          "TargetGroups": [
             {"TargetGroupArn": "<Blue Target Group>", "Weight": 0}, \
             {"TargetGroupArn": "<Green Target Group>", "Weight": 100}, \
          ]
       }
    }]'

The following diagram shows the updated architecture.

Figure 5: Blue-green deployment architecture with 0% blue traffic and 100% green traffic

If you need to enable target group level stickiness, you can ensure that all traffic transitions from the blue target group to the green target group by keeping the target group level stickiness duration as low as possible (5 minutes or less).

In the following code, the target group level stickiness is enabled for a duration of 5 minutes and traffic is completely shifted from the blue target group to the green target group:

aws elbv2 modify-listener \
    --listener-arn "<LISTENER ARN> " \
    --default-actions \
    '[{
       "Type": "forward",
       "Order": 1,
       "ForwardConfig": {
          "TargetGroups": [
             {"TargetGroupArn": "<Blue Target Group>", "Weight": 0}, \
             {"TargetGroupArn": "<Green Target Group>", "Weight": 100}, \
          ],
          "TargetGroupStickinessConfig": {
             "Enabled": true,
             "DurationSeconds": 300
          }
       }
    }]'

The existing users with connection stickiness to the blue target group continue to the blue target group until the 5-minute duration elapses from the last request time.

As recommended above, the recommended value of stickiness may vary across application types.

Connection draining

To provide near-zero downtime release with blue/green deployment, you want to avoid breaking open network connections while taking an instance out of service, updating its software, or replacing it with a fresh instance that contains updated software. In the above use cases, you can ensure graceful transition between blue and green target groups by enabling the connection draining feature for your Elastic Load Balancers. You can do this from the AWS Management Console, the AWS CLI, or by calling the ModifyLoadBalancerAttributes function in the Elastic Load Balancing API. You can enable the feature and enter a timeout between 1 second and 1 hour. The connection time out duration depends upon your application profile. If your application is stateless like your customers are using your website, connection time out duration of lowest value is preferable. Applications that are transactions heavy and connection oriented sessions like web sockets, we recommend you to choose relatively high connection draining duration as it will impact the customer experience adversely.

Load balancer stickiness

In addition to the target group level stickiness, Application Load Balancer also supports load balancer level stickiness. When a load balancer first receives a request from a client, it routes the request to a target, generates a cookie named AWSALB that encodes information about the selected target, encrypts the cookie, and includes the cookie in the response to the client. The client should include the cookie that it receives in subsequent requests to the load balancer. When the load balancer receives a request from a client that contains the cookie, if sticky sessions are enabled for the target group and the request goes to the same target, the load balancer detects the cookie and routes the request to the same target. If the cookie is present but can’t be decoded, or if it refers to a target that was deregistered or is unhealthy, the load balancer selects a new target and updates the cookie with information about the new target.

You can enable Application Load Balancer stickiness using the AWS CLI or the console. You can specify a value between 1 second–7 days.

In the context of blue/green and canary deployments, the load balancer stickiness has no influence on the traffic shifting behavior using the weighted target groups because target group stickiness takes precedence over load balancer stickiness.

Conclusion

In this post, we showed how to perform canary and blue/green deployments with Application Load Balancer’s weighted target group feature and how target group level stickiness impacts your canary and blue/green deployments. We also demonstrated how quickly you can enable ELB connection draining to provide near-zero downtime release with blue/green deployment. We hope that you find these recommendations helpful when you build a blue/green deployment with Application Load Balancer. You can reach out to AWS Solutions Architects and AWS Support teams for further assistance.

Raghavarao Sodabathina is an Enterprise Solutions Architect at AWS, focusing on Data Analytics, AI/ML, and Serverless Platform. He engages with customers to create innovative solutions that address customer business problems and accelerate the adoption of AWS services. In his spare time, Raghavarao enjoys spending time with his family, reading books, and watching movies.

Siva Rajamani is a Boston-based Enterprise Solutions Architect for AWS. He enjoys working closely with customers, supporting their digital transformation and AWS adoption journey. His core areas of focus are Serverless, Application Integration, and Security. Outside of work, he enjoys outdoor activities and watching documentaries.

TAGS: blue-green deployments

AWS DevOps & Developer Productivity Blog