AWS Cloud Operations Blog

Combating Configuration Drift Using Amazon EC2 Systems Manager and Windows PowerShell DSC

Configuration drift occurs when a system “drifts” or changes from its intended configuration. It is caused by having inconsistent configuration items (CIs) across environments.

Amazon EC2 Systems Manager is a management service that helps you automatically collect a software inventory, apply OS patches, create system images, and configure Windows and Linux operating systems. These capabilities help you define and track system configurations, prevent drift, and maintain software compliance of your EC2 and on-premises configurations.

Systems Manager provides a management approach that is designed for the scale and agility of the cloud but extends into your on-premises data center. Systems Manager makes it easier for you to seamlessly bridge your existing infrastructure with AWS.

In my last post, I introduced the concept of using Systems Manager Run Command to apply a declarative based model for EC2 instance configuration (configuration as code) via Windows PowerShell Desired State Configuration (DSC). In this post, I show how you can combat configuration drift at scale using PowerShell DSC and a management tool from Systems Manager called State Manager.

Configuration drift

Configuration drift can happen for a multitude of reasons and can occur in many types of environments such as database systems, network configurations, directory services, web servers, and custom APIs. Some possible reasons that drift can occur include system updates, code pushes, hardware upgrades, or software updates that get applied to some, but not all of your intended systems. I am sure that you have seen this firsthand.  We have all heard “but it worked in my dev environment.”

Configuration drift can cause problems that can result in unintended system behaviors, system failures, disaster recovery issues, and many other potential problems. Fortunately, there are tools and technologies that help mitigate drift.

  • Configuration as code

Declaratively define a desired state within a configuration file that can be applied to a system to interpret and apply.

  • State Manager

An EC2 tool that helps you apply a configuration to a set of on-premises servers and EC2 instances on a scheduled basis.

  • Configuration management databases (CMDB)

CMDBs help track the state of IT assets (also known as configuration items), allowing IT administrators the ability to check on the current state of CIs at any time. A common issue with CMDBs is stale data, meaning that the current state is not always up-to-date. The inability to reliably detect drift is problematic as mitigation is not performed. While CMDBs are popular, I do not use them as a solution in this post.

Configuration as code via PowerShell DSC

Before I dive into a solution to help with the ongoing struggles of configuration drift, here are the benefits of using configuration as code via PowerShell DSC:

  • Definition files.

A system “desired state” is defined in declaratively based configuration files. You no longer need to develop custom code and scripts to handle the configuration state that you want for your servers.

  • Idempotency.

You can apply the configuration file repeatedly without adverse side effects.

  • Self-documentation.

The configuration files are human-readable and easily understood.

State Manager

State Manager allows you to schedule a script to be run against your on-premises servers and EC2 instances on a scheduled basis. If the scripts are idempotent, you can apply them repeatedly. This gives way to a simple and scalable pattern that helps mitigate configuration drift.

  1. Define your set of servers.
  2. Create idempotent scripts or use PowerShell DSC to define your desired state configuration.
  3. Apply the script on a scheduled basis to mitigate any drift that may have occurred since the last time you ran the script.

State Manager and PowerShell DSC

This is a powerful combination! State Manager and PowerShell DSC allow you to define configuration policies that can be applied to your EC2 instances at scale. It ensures that your instances remain in that state by reapplying the policy on a defined schedule. While this does not prevent drift from occurring, it helps you mitigate any drift that may have occurred. Thus, this is an excellent pattern to ensure that both your on-premises servers and EC2 instances stay in your desired state.

In this section, I show you how to use State Manager from the AWS Management Console. You learn to use both State Manager and PowerShell DSC to combat configuration drift.

At a high level, this is the walk-through process:

  1. Create EC2 Windows instances.
  2. Tag those instances such that they are considered a set of instances that share a common purpose.
  3. Create a State Manager association. This association executes a PowerShell DSC script against the tagged instances on a specified schedule.
    • The PowerShell DSC script is contained in a PowerShell module downloaded from a S3 endpoint. It is authored by AWS and its intent is to install IIS on the instance and enable ASP and Tracing features. The module is publicly available and can be downloaded.

Walk-through

  1. In the EC2 console, create two or more new Windows Server instances.
  2. Choose Instances, select the instances that you just created, and choose Tags.
  3. Choose Add/Edit Tags and set the following values:
    • Set Key = Metadata
    • Set Value = {“Environment”: “Production”, “Type”: “IIS Web Server”}
  4. In the EC2 console, choose State Manager, Create Association, and select the document AWS-InstallPowerShellModule. Set the following values:
    • Keep the document version as $DEFAULT.
    • For Specifying a Tag, select:
      • Tag Name = Metadata
      • Tag Value = {“Environment”: “Production”, “Type”: “IIS Web Server”}
    • For Schedule, choose Every 30 Minutes.
    • For the Source, enter the following URL: https://s3.amazonaws.com/aws-windows-samples-us-east-1/PSModules/SSMDevOps.zip
    • For Commands, copy and paste the following:
    • Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Force
Import-Module SSMDevOps
Install-SsmDoIIS
Start-DscConfiguration -Path Install-SsmDoIIS -Wait
  5. Choose Create Association.

Conclusion

Systems Manager offers a suite of tools to help you manage both your EC2 and on-premises instances. In this post, I discussed some common approaches to mitigate configuration drift at scale. Finally, I provided an example walk-through to try mitigating configuration drift using Systems Manager and PowerShell DSC.

About the Author

Shaun Breen is a Systems Development Engineer on the Amazon EC2 Windows team. The EC2 Windows team is responsible for producing Windows Server AMIs and Systems Manager documents. Shaun enjoys developing solutions that make EC2 the best place to run Microsoft Windows Server in the cloud. When not working on EC2 Windows solutions, he enjoys attending sporting events, coaching ice hockey, and spending time with his wife and three children.