AWS Cloud Operations Blog

Scalable cross-platform patching with AWS Systems Manager

Guest Post by Jakub Marciniak, Joseph Bulluss, Mariusz Borys, Thushyanth Sivananthan (DevOps Engineers @Infor), David Benjamin(Lead @Infor) and Martin Jarosinski(Manager @ Infor).

————————————————————————————————————————————————————————————

At Infor, in our effort for continual improvement, we were researching tools that would allow for more efficiency in our patching and overall system management. When the Meltdown and Spectre vulnerabilities were announced, they presented Infor with some unique challenges and an opportunity. Addressing these vulnerabilities would involve multiple products, OS platforms, software patches, registry changes, hypervisors, and anti-virus products, so this was a perfect time to improve upon our OS patching tooling using our prior tooling research. We strongly wanted agent-based management software that would work within the AWS ecosystem relatively seamlessly and help us manage and patch over 15,000 dedicated, and privately network-segmented, customer servers.

Our testing showed that AWS Systems Manager was a perfect fit, so we decided to leave our older tools behind and move forward. Furthermore, we found that OS patching was considerably faster with AWS Systems Manager. Our previous tools could take up to 90 minutes, while AWS Systems Manager completes its tasks within 30 minutes on average. This translates into shorter maintenance windows. With proper tagging in place, newly deployed instances can be targeted easily to be included in our patching solution.

Our first step was to configure a default patch baseline for Windows. I’ll walk you through how we did it.

  1. Open the AWS Systems Manager console at https://console.thinkwithwp.com/systems-manager/.
  2. In the navigation pane, choose Patch Manager.
  3. To create your own default patch baseline, in the navigation pane, choose Patch Manager, and then choose Create patch baseline.

From here we created a patch baseline and set the approval rules to Operating System: Windows, Product: All, Classification: SecurityUpdates, and left Auto approval delay to 0 days.

We then set this baseline as the default patch baseline under Actions in Patch Manager.

Next, we created an SSM Document. Documents control the actions you want to perform on your instances. AWS Systems Manager schema version 2.2 allows for cross-platform support allowing both Windows and Linux Documents to be combined into one Document for ease of maintainability. We used the Document “AWS-RunPatchBaseline,” which is provided by AWS. We made some modifications which included required registry changes to solve Meltdown and Spectre vulnerabilities. Here’s what we did:

1. Registry changes for Trend Micro anti-virus.

"$TrendMicroVersion = Get-WmiObject -Class Win32_Product -Filter {Name = 'Trend Micro Deep Security Agent'} | Select-Object -ExpandProperty Version",

          "if (($TrendMicroVersion -like '9.6*') -or ($TrendMicroVersion -like '10*') -or ($TrendMicroVersion -eq $null)) {",

          "'Applying registry key cadca5fe-87d3-4b96-b7fb-a231484277cc...'",

          "reg add 'HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\QualityCompat' /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0 /f",

          "start-sleep 1",

          "}"


2.   Registry keys update to enable the mitigations on the server.

"'Applying registry key FeatureSettingsOverride...'",

          "reg add 'HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management' /v FeatureSettingsOverride /t REG_DWORD /d 1 /f",

          "start-sleep 1",

          "'Applying registry key FeatureSettingsOverrideMask...'",

          "reg add 'HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management' /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f",

          "start-sleep 5",

3.     Support for PowerShell 2.0 and .Net 4.0.

"function LoadDotNet4 {",

          "    $file = \"$($env:WINDIR)\\System32\\WindowsPowerShell\\v1.0\\powershell.exe.config\"",

          "    if (Test-Path ($file)) {",

          "        Rename-Item \"$file\" \"$($file)_bak\" -ErrorAction SilentlyContinue",

          "    }",

          "    '<?xml version=\"1.0\"?>' | Out-File $($file)",

          "    '<configuration>' | Out-File $($file) -Append",

          "    '  <startup useLegacyV2RuntimeActivationPolicy=\"true\">' | Out-File $($file) -Append",

          "    '    <supportedRuntime version=\"v4.0.30319\"/>' | Out-File $($file) -Append",

          "    '    <supportedRuntime version=\"v2.0.50727\"/>' | Out-File $($file) -Append",

          "    '  </startup>' | Out-File $($file) -Append",

          "    '</configuration>' | Out-File $($file) -Append",

          "}",

          "if ($($PSVersionTable.PSVersion.Major) -le 2) {",

          "   LoadDotNet4",

          "}"

The Windows patching part of the Document was left untouched because the default AWS SSM Document gave us all we needed. For Linux, it was simplified in that we did not want to apply all patches, but just the required kernel update. We modified the Linux portion of the Document that checks for the flavor of Linux and updates the kernel version accordingly. A section of the Document also checks to see if the instance is PV type virtualization and skips patching accordingly. This is important for Linux environments since the PV instances will not boot up again after patching due to a bug with the Meltdown and Spectre kernel updates.

While it’s possible to set patching to occur on a schedule using Maintenance Windows, our operations required us to run the Document on demand. With the Document complete, our first round of testing was done directly from the AWS Systems Manager console using Run Command. We targeted instances by instance ID.

1.   Open the AWS Systems Manager console at https://console.thinkwithwp.com/systems-manager/.

2.   In the navigation pane, choose Run Command.

3.   Select the Command Document.

4.   Select targets by manually or by tag.

5.   Configure parameters.

6.   Configure Output options. Specify Amazon S3 bucket if required.

Finally, choose Run and take note of the Command ID. You can monitor status in Run Command. Once status is successful you can review the logs by highlighting the Command ID and choosing the Output tab. Any problems that arise can be investigated by checking the output or the SSM logs on the instance.

On Windows

%PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
%PROGRAMDATA%\Amazon\SSM\Logs\error.log

On Linux

/var/log/amazon/ssm/amazon-ssm-agent.log
/var/log/amazon/ssm/errors.log

For Windows, you can check if all assigned patches have been installed in Patch Compliance.

Once we confirmed the Document was working as expected we went a step further and scripted execution using Python and Boto3. The core of the script is the AWS CLI command which is output when setting up in Run Command, with the addition of flexible targeting, error checking, and the ability to work with multiple accounts and multiple AWS Regions in a single execution.

Our script makes the following Boto3 API calls:

1.     ec2.describe_instances – Pull and verify list of instances based on input CSV file, VPC id, instance Id or EC2 tags.

2.     ssm.describe_instance_information – Verify if SSM Agent is installed/Online and compare with list gathered in point 1.

3.     s3.list_buckets – Find correct S3 bucket to store SSM command logs.

4.     ssm.send_command – Execute SSM Document against groups of instances within specific region and store logs to S3 bucket.

5.     ssm.list_command_invocations – Verify status for each commandId execution per SSM Agent.

6.    ssm.list_resource_compliance_summaries – Pull compliance report for all SSM Agents within specific Region to get patch state for each agent.

Conclusion

We found AWS Systems Manager exceedingly straightforward and easy to set up. We’ve now started exploring some of its other features. State Manager Associations will allow us to enforce policy and compliance across all our instances. Inventory Manager, which only requires the SSM agent and an existing Document from AWS, allows us to collect and manage inventory data such as installed applications, AWS components, patches, files, Windows registry keys, and custom inventory.

The content and opinions in this post are those of  third-party authors and AWS is not responsible for the content or accuracy of this post.

Authors

DevOps Engineers: Jakub Marciniak, Joseph Bulluss, Mariusz Borys, Thushyanth Sivananthan
David Benjamin – Lead
Martin Jarosinski – Manager