Integrating Network Connectivity Testing with Infrastructure Deployment

This post was co-authored with Eduardo Janicas

In this blog post, we will show how to integrate and automate network connectivity testing as part of AWS infrastructure deployment. We will show you how to use a new solution, Infrastructure as Code (IaC) Network Tester, to perform connectivity testing between a source resource and a destination resource in Virtual Private Cloud (VPC). It supports testing resources deployed via AWS Management Console, SDK, Command Line Interface (CLI) or via a CI/CD pipeline.

Background

AWS provides a set of flexible Infrastructure as Code services and framework designed to enable companies to more rapidly and reliably build and deploy infrastructure. For example, AWS cloud resources can be created using AWS CloudFormation, AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK) or other third party tools like Terraform or Serverless. These services simplify provisioning and managing infrastructure, deploying application code and automating software release processes.

When you define your infrastructure template, you have full control over your virtual network environment. You choose your own IP address range, create subnets, configure route tables and gateways. You also add multiple layers of security, such as security groups and network access control lists (NACLs), to control access to entities of each subnet by protocol, IP address, and port number.

One of the challenges customers face is implementing the minimum set of configurations to allow traffic to flow through network paths as intended. As your infrastructure grows, it is common for these templates to have thousands of lines of code. With the growth of complexity, it is not unusual to have unintended configuration that could lead to network paths that are not reachable or establish new network paths which are not intended.

In re:Invent 2020, AWS launched the VPC Reachability Analyzer. This is a configuration analysis tool that enables you to perform connectivity testing between a source and destination resource in a VPC. By integrating the VPC Reachability Analyzer with infrastructure deployment, you can troubleshoot connectivity issues caused by misconfiguration, and when integrated into a continuous integration and continuous (CI/CD) pipeline, you can proactively verify during the testing phases that your configuration matches your network connectivity intent.

Overview of the IaC Network Tester

The IaC Network Tester is a state machine powered by AWS Step Functions that helps you run network testing against a set of source and destination resources to ensure configuration matches intent. Figure 1.0 below shows a high-level overview of the tester.

Figure 1.0: High Level Overview of Integrating Network Testing to Infrastructure Deployment

Infrastructure created using any infrastructure as code tool will need to output a JSON formatted string specifying the source, destination and tag for each of the route to be tested. For stacks deployed via AWS Console, SDK or CLI, the tester takes that output value of the stack as its input and then runs post infrastructure deployment to validate the network paths created in your topology. If the infrastructure is deployed via a CI/CD pipeline, the tester can be integrated into the testing phase of the pipeline prior to deploying to production.

The IaC Network Tester executes the network path testing by running a step function that carries out the following steps:

1. Identify the routes to be tested by retrieving the JSON formatted output from the stack output
2. Start the network test by invoking VPC Reachability Analyzer concurrently for the routes
3. Wait for the test to run and retrieve the test results
4. Clean up the VPC Reachability Analyzer resources used to carry out the test
5. Repeat the process from Step 2 if more than five routes are to be tested (tests are done in batches of 5 due to the quota for concurrent analyses for VPC Reachability Analyzer)

The output from the state machine indicates which routes are reachable, which are unreachable and reason for unreachable routes. For infrastructure deployed via Console, SDK or CLI, the results of these tests help you to correct the configuration issues and redeploy the stack. For infrastructure deployed via CI/CD pipeline, this result can be used to ensure the configuration matches intent and if not fail the pipeline and prevent deploying into production.

Walkthrough of using IaC Network Tester

To demonstrate the IaC Network Tester, we will deploy a sample infrastructure consisting of a multi-VPC architecture with VPC peering. In this architecture a shared VPC is used to provide Active Directory Services to a Remote Desktop Service VPC and Business Application VPC as shown in Figure 1.1 below. These two VPCs contain a public subnet that provide users access to the services via the Internet. To authenticate, authorize and manage users, the VPCs are connected to the Active Directory Service VPC via a peering connection. The security group of the servers within each VPC is configured as follows:

• Remote Desktop Server Security Group allows inbound traffic on port 3389 from the internet and outbound traffic on port 53 to the Active Directory Service VPC
• Business Application Server Security Group allows inbound traffic on port 80 form the internet and outbound traffic on port 53 to the Active Directory Service VPC
• Active Directory Server Security Group allows inbound traffic on port 53 from the public subnets in the Remote Desktop Services VPC and Business Application VPC. This VPC is not required to have internet access and also does not need to initiate traffic to the other VPCs.

Figure 1.1: Sample infrastructure for a Shared Services VPC using VPC Peering

As part of deployment of this infrastructure to your AWS Account, the IaC Network Tester can be used to validate that these configurations match intent. Some of the network paths in this architecture which can be tested are given below:

Network Paths which are intended to be reachable:

1. Remote Desktop instance to Active Directory instance
2. Business Application instance to Active Directory instance
3. Internet to Business Application instance
4. Internet to Remote Desktop instance

Network Paths which are intended NOT to be reachable:

5. Business Application Instance to Remote Desktop instance
6. Active Directory instance to Business Application instance
7. Active Directory instance to Remote Desktop instance
8. Remote Desktop instance to Internet

The steps required to use the solution for these two scenarios include:

Scenario 1 – Using IaC Network Tester on Infrastructure Deployed using CloudFormation via Console, CLI or SDK

1. Deploy the IaC Network Tester application on your AWS account
2. Deploy the CloudFormation template for the sample infrastructure
3. Initiate the IaC Network Tester state machine

Scenario 2 – Integrating IaC Network Tester into a CI/CD Pipeline

1. Deploy the IaC Network Tester application on your AWS account
2. Deploy the provided CloudFormation template for a sample CI/CD pipeline built using AWS CodePipeline
3. Upload a zip file containing the CloudFormation template of the sample infrastructure to the CI/CD pipeline source S3 bucket
4. Trigger the pipeline which will invoke the IaC Network Tester state machine during the network testing stage

Prerequisites

• An AWS Account
• AWS CloudShell in your AWS Account (recommended) or you can use a command line tool within your local system which will require SAM CLI. Follow the instructions here to install SAM CLI.
• Access to the GitHub Repository containing the IaC Network Tester and the sample stacks

Scenario 1 – Using IaC Network Tester on Infrastructure Deployed via Console, CLI or SDK

Step 1: Deploy the IaC Network Tester application on your AWS account

From your AWS CloudShell terminal (you could, optionally, set up your local machine to do this by installing the SAM CLI), clone the IaC Network Tester repository using the following example commands below:

git clone https://github.com/aws-samples/aws-iac-network-tester.git
cd aws-iac-network-tester
sam build
sam deploy --guided

The “sam deploy —guided” command prompts you to specify a number of parameters to deploy the SAM application. You can select the defaults by pressing the “Enter” button for each prompt as show figure 1.2 below.

Figure 1.2: Default options for Deploying the IaC Network Tester SAM Application

On successful completion of that command, you see the output similar to what is shown in figure 1.3.  Note the value of the key IaCNetworkTesterStateMachineArn. This is the Amazon Resource Name (ARN) of the IaC Network Tester state machine.

Figure 1.3: Output from Deploying the IaC Network Tester SAM Application

Step 2: Deploy the CloudFormation template for the sample infrastructure

This solution provides sample CloudFormation template sample-stack-multi-vpc.yml for creating the network infrastructure shown in figure 1.1. The sample CloudFormation template is located in the folder sample_resources within the cloned repository. To enable the IaC Network Tester to test network resources deployed by the CloudFormation stack, the routes to be tested will have to be specified as one of the outputs of the stack. In the sample stack provided, find below the stack output “NetworkReachabilityTestPaths” which specifies the routes to be tested.

# Routes to test by IaC Network Tester
Outputs:
  NetworkReachabilityTestPaths:
    Value: !Sub |
      [
      {"Source":"${RemoteDesktopInstance}", "Destination":"${ActiveDirectoryInstance}","RouteTag":"RemoteDesktopToActiveDirectory"},
      {"Source":"${BusinessAppInstance}", "Destination":"${ActiveDirectoryInstance}","RouteTag":"BusinessAppToActiveDirectory"},
      {"Source":"${BusinessAppVPCIG}", "Destination":"${BusinessAppInstance}","RouteTag":"InternetToBusinessApp"},
      {"Source":"${BusinessAppInstance}", "Destination":"${RemoteDesktopInstance}","RouteTag":"BusinessAppToRemoteDesktop"},
      {"Source":"${ActiveDirectoryInstance}", "Destination":"${BusinessAppInstance}","RouteTag":"ActiveDirectoryToBusinessApp"},
      {"Source":"${ActiveDirectoryInstance}", "Destination":"${RemoteDesktopInstance}","RouteTag":"ActiveDirectoryToRemoteDesktopApp"},
      {"Source":"${RemoteDesktopInstance}", "Destination":"${RemoteDesktopVPCIG}","RouteTag":"RemoteDesktopToInternet"},
      {"Source":"${RemoteDesktopVPCIG}", "Destination":"${RemoteDesktopInstance}","RouteTag":"InternetToRemoteDesktopApp"}
      ]

This is a JSON formatted array that specifies the routes to test with each item on the array containing the following keys:

• Source – The identifier of the source resource where the traffic will originate.
• Destination – The identifier of the destination resource where the traffic will terminate
• RouteTag – An identifier for the route between source and destination which needs to be validated. This can be specified with a user friendly name.

To use the solution with your infrastructure template, follow a similar format for the output and modify the JSON to contain the routes within your stack for testing. Create the sample infrastructure shown in figure 1.1 above by deploying the CloudFormation template using the following example command.

cd sample_resources
aws cloudformation create-stack --stack-name iac-network-tester-sample-stack \
--template-body file:///$PWD/sample-stack-multi-vpc.yml \
--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

Step 3 – Initiate the IaC Network Tester state machine

On successfully deployment of the sample stack, initiate the IaC Network Tester state machine by starting a new execution using the command below, providing values for the parameters.

aws stepfunctions start-execution \
--state-machine-arn <state_machine_arn> \
--input "{\"stackName\": \"<sample_stack_name>\", \"routeToTestOutputKey\": \"<route_to_test_output_key>\", \"analysisDuration\": 15, \"analysisWaitCount\": 3}"

The <state_machine_arn> parameter is the value from Step 1 above. The input to the state machine consists of:

• stackName – The name of the CloudFormation stack to test for e.g. “iac-network-tester-sample-stack”
• routeToTestOutputKey – This represents the output key in the Cloudformation stack containing the paths to test. In the sample stack provided, the value would be “NetworkReachabilityTestPaths”.
• analysisDuration – The duration in seconds which specifies the time to wait for the VPC Reachability Analysis to run after initiating the analysis. The analysis can take up to several minutes depending on the size and complexity of your VPCs, but it typically takes a few seconds. This functionality uses AWS Step Function Wait state.
• analysisWaitCount – The number of times to wait for the analysis to run if after the analysisDuration the test is still running. Each wait is the duration specified in analysisDuration.

Note the executionArn returned after running the command as shown in figure 1.4 below

Figure 1.4: Output from initiating the IaC Network Tester Step Function

The output from the state machine contains the details of the test result and can be retrieved by running the command below or looking at the state machine output via the console:

aws stepfunctions describe-execution \
--execution-arn "<execution_arn>"

A sample output from the IaC Network Tester state machine is shown in figure 1.5 below. It is a JSON formatted string that shows the tests that succeeded, timed out (if the analysis did not complete within the configured time) or failed. If some tests timed out, the test can be rerun with an increased “analysisDuration” or “analysisWaitCount” to give the test more time to complete. The results provide following information for each route tested.

Figure 1.5: IaC Network Tester Output for Sample Infrastructure

For each route tested, the results show the Source, Destination and the RouteTag which were specified in the input. Apart from these, additional parameters such as NetworkInsightsPathId, NetworkInsightsAnalysisId, are identifiers of internal analysis objects created. NetworkPathFound and Explanations are details from the VPC Reachability Analysis of each route. NetworkPathFound indicates if the route is reachable and if not the Explanations field provides details of why the route is not reachable. For more details about these two fields refer to the documentation. The architecture diagram of the sample infrastructure in figure 1.6 below shows the eight routes tested and the routes that are reachable (in green) and NOT reachable (in red) which validates that the configuration matches intent.

Figure 1.6: Sample infrastructure showing the five routes tested and the routes that are Reachable and NOT Reachable.

 Scenario 2 – Integrating IaC Network Tester into a CI/CD Pipeline

IaC Network Tester can be triggered from a CI/CD pipeline that deploys an infrastructure stack. To show how this works, we will be deploying the sample CI/CD pipeline shown in figure 1.7 below used to deploy the sample infrastructure shown in figure 1.1.

Figure 1.7: CI/CD Pipeline for deploying CloudFormation template invoking the IaC Network Tester

In the Test Stage, one of the tests that can be conducted is to validate the reachability of routes defined within the template. The “Execute Network Test” phase invokes the IaC Network Tester and based on the results of the test can proceed or fail the pipeline execution. The steps below will deploy the sample CI/CD pipeline and show how the IaC Network Tester can integrate with the pipeline.

 Step 1: Deploy the IaC Network Tester application on your AWS account

Follow the instruction in Step 1 of Scenario 1 to deploy the IaC Network Tester on your AWS account. Note the execution_arn of the step function returned after deploying the IaC Network Tester application. Note: You will need to delete the sample stack created in Scenario 1 above if the quota for number of VPCs in the region you are working on is the default value of 5. The CI/CD pipeline will create the same sample infrastructure which will fail if the number of VPCs exceeds 5.

Step 2: Deploy the CloudFormation template for the sample CI/CD pipeline

The sample pipeline template sample-pipeline.yml is located in the folder sample_resources within the cloned repository. This will deploy a CI/CD pipeline using AWS CodePipeline and can be used as the CI/CD pipeline to deploy the sample infrastructure defined in sample-stack-multi-vpc.yml. The key addition to the CI/CD pipeline is a phase that executes network testing post deployment of the infrastructure in a test environment. The code snippet below shows the configuration of this phase which invokes the IaC Network Tester Step Function. For your existing CI/CD pipelines, this new action can be included in the test stage while providing the parameters required to invoke the IaC Network Tester in the CI/CD pipeline template parameters.

- Name: ExecuteNetworkTest
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Provider: StepFunctions
                Version: 1
              Configuration:
                StateMachineArn: !Ref IaCNetworkTesterStateMachineArn
                ExecutionNamePrefix: iac-network-tester
                Input: !Join
                  - ""
                  - - '{"stackName":"'
                    - !Ref TestStackName
                    - '",'
                    - '"routeToTestOutputKey": "'
                    - !Ref IaCNetworkTesterRouteToTestOuputKey
                    - '",'
                    - '"analysisDuration": '
                    - !Ref IaCNetworkTesterAnalysisDuration
                    - ","
                    - '"analysisWaitCount": '
                    - !Ref IaCNetworkTesterAnalysisWaitCount
                    - "}"
              OutputArtifacts:
                - Name: networkTestOutput
              RunOrder: "2"

To create the sample pipeline, deploy the CloudFormation template using the following command below, replacing the parameters:

• myEmailAdd – used for pipeline SNS notification messages. Post deploying the CI/CD pipeline a confirmation email will be sent to this address which you will need to confirm to complete the SNS subscription.
• IaCNetworkStateMachineArn – the ARN of the IaC Network Tester as returned from Step 1 above
• routeToTestOutputKey – This represents the output key in the Cloudformation stack containing the paths to test. In the sample stack provided, the value would be “NetworkReachabilityTestPaths”.
• analysisDuration – The duration in seconds which specifies the time to wait for the VPC Reachability Analysis to run after initiating the analysis. The analysis can take up to several minutes depending on the size and complexity of your VPCs, but it typically takes a few seconds. This functionality uses AWS Step Function Wait state.
• analysisWaitCount – The number of times to wait for the analysis to run if after the analysisDuration the test is still running. Each wait is the duration specified in analysisDuration.

cd sample_resources
aws cloudformation create-stack --stack-name iac-network-tester-sample-pipeline \
--template-body file:///$PWD/sample-pipeline.yml \
--parameters ParameterKey=Email,ParameterValue=<myEmailAdd> \
ParameterKey=IaCNetworkTesterStateMachineArn,ParameterValue=<IaCNetworkStateMachineArn> \
ParameterKey=IaCNetworkTesterRouteToTestOuputKey,ParameterValue=<routeToTestOutputKey> \
ParameterKey=IaCNetworkTesterAnalysisDuration,ParameterValue=<analysisDuration> \
ParameterKey=IaCNetworkTesterAnalysisWaitCount,ParameterValue=<analysisWaitCount> \
--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

Step 3: Upload a zip file containing the CloudFormation template of the sample infrastructure to the source S3 bucket

The sample CI/CD pipeline is configured to use S3 as the source artifact however IaC Network Tester can be used with other sources supported by AWS CodePipeline. To configure the source details for the pipeline, the sample-pipeline.yml contains following parameters:

• SourceS3Key: which is used to specify the S3 object key for the zip file containing the cloudformation template to deploy by the pipeline. It defaults to iacnetworktestersamplestack.zip.
• TemplateFileName: which is used to specify the cloudformation template file name to deploy and contained within the zip file. It defaults to sample-stack-multi-vpc.yml.

The sample-pipeline.yml also creates the S3 bucket that will contain the source artefacts with the name format “${AccountID}-iac-nt-bucket”. To use a different naming format, update the “SourceBucket” resource within the cloudformation template. Use the sample command below to create a zip file containing the sample stack template file and upload the file to the S3 bucket.

cd sample_resources
zip iacnetworktestersamplestack.zip sample-stack-multi-vpc.yml
aws s3 cp iacnetworktestersamplestack.zip s3://<AccountID>-iac-nt-bucket/

Step 4: Trigger the pipeline which will invoke the IaC Network Tester state machine during the network testing stage

Uploading the to the S3 bucket should trigger the pipeline automatically or you can also trigger the pipeline by executing the command below specifying the pipeline name. This is one of the parameters to the sample-pipeline.yml template which has a default value of iac-network-tester-sample-pipeline.

aws codepipeline start-pipeline-execution –name iac-network-tester-sample-pipeline

You can navigate to the AWS CodePipeline console to visualise the pipeline execution as shown in Figure 1.8 below. The pipeline retrieves the sample stack template from the S3 bucket within the S3Source stage and then moves to the TestStage. Within the TestStage, the pipeline creates the test stack in the CreateStack phase and then moves to the ExecuteNetworkTest phase where it invokes the IaC Network Tester. If this completes successfully, the pipeline then moves to the ApproveTestStack phase where the admin can review the output from the step function and ensure that configuration matches intent. This verification process can also be automated by invoking a AWS Lambda function to analyse the test results and if configuration matches intent the pipeline can continue to the next phase, otherwise the pipeline will fail.

Figure 1.8 CodePipeline Execution with Integration to IaC Network Tester

IaC Network Tester Costing

There are three main costs associated with IaC Network Tester which will only be incurred when the step function is invoked. The AWS pricing below will be based on the North Virginia region:

• AWS Step Functions cost: corresponds to the number of state transitions during the state machine execution. This cost depends on the number of routes tested and test duration. For example, if 10 routes are to be tested and all the test completed within the analysisDuration variable provided. The cost will be approximately $0.0005.
• AWS Lambda cost: related to the Lambda function executed at the following states of the step function: IntializeTest, RetrievePathToTest, StartNetworkTest, CheckTestStatus and CleanUp. On testing 10 routes with 128MB memory configuration for the lambda functions, the total duration was 120,000 milliseconds which will cost $0.000252.
• Amazon VPC Reachability Analyzer cost: corresponds to charge for each connectivity analysis between a given source and destination at $0.10 per analysis. If you test 10 routes, the total cost will be $1.

Cleaning up

Leaving resources that you don’t need running on your AWS account will incur changes. Follow the instructions below to clean up the resources created in this post and avoid incurring further charges.

 Scenario 1 – Using IaC Network Tester on Infrastructure Deployed via Console, CLI or SDK

1. Run the following CLI commands to delete the CloudFormation stacks.

aws cloudformation delete-stack --stack-name iac-network-tester-sample-stack
aws cloudformation delete-stack --stack-name iac-network-tester-app

Scenario 2 – Integrating IaC Network Tester into a CI/CD Pipeline

1. Delete the CodePipeline Artifact Store S3 Bucket (iac-network-tester-sample-pip-artifactstorebucket-XXXXXXXXXXXXX) and Source S3 Bucket (<AccountID>-iac-nt-bucket) via the console by emptying the bucket and then deleting the bucket.
2. Run the following CLI commands to delete the CloudFormation stacks. Ensure each command successfully deletes the resources before proceeding to the next

aws cloudformation delete-stack --stack-name iac-network-tester-sample-stack
aws cloudformation delete-stack --stack-name Prod-IaCNetworkTesterEnv
aws cloudformation delete-stack --stack-name iac-network-tester-sample-pipeline
aws cloudformation delete-stack --stack-name iac-network-tester-app

Conclusion

In this blog post, we have covered how you can identify network misconfiguration in your infrastructure deployed using Infrastructure as Code tools. While the examples in this blog used IaC Network Tester with Infrastructure deployed through CloudFormation and AWS Code Pipeline, the same approach can be incorporated in other IaC tools or CI/CD tools to automate the network testing. This will help ensure that network configurations in a VPC match intent and any misconfiguration can be identified prior to deploying to production. You can read more about VPC Reachability Analyzer and the resources that are currently supported as the source and destination. We welcome your feedback and if you have any improvements, new feature requests or bugs kindly raise them via the issues tab on GitHub. Happy Testing!