Open sesame: Granting privileged access to EC2 instances with Session Manager

In this guest blog post, Herman Lee (Cloud Solution Architect, VP) and Nauman Noor (Managing Director) from the public cloud engineering team at State Street discuss their use of AWS Systems Manager Session Manager for privileged access management of Amazon EC2 instances.

State Street Corporation is a financial services company responsible for the management, custody, and administration of trillions of dollars of financial assets worldwide. As one of the oldest and largest banks in the world, State Street attempts to stay at the forefront of technology and invests heavily in cloud technologies.

To support situations where application teams or third-party vendors need access to an Amazon Elastic Compute Cloud (Amazon EC2) instance to diagnose a production issue or to make emergency changes during a firecall, State Street needed to support a set of capabilities generally known as Privileged Access Management (PAM). These capabilities include providing administrators with elevated permissions to the EC2 instances in an approved timeframe and auditing functions such as access and keystroke logging.

Why State Street chose Session Manager for PAM

The team looked at commercial PAM offerings and found that they are often costly and involve complex account and password management of the privileged accounts, which in our case are the operating system user accounts on EC2 instances. These commercial PAM offerings often rely on direct network access to the EC2 instances and require deep integration efforts with existing logging infrastructure. These shortcomings gave us the impetus to build our PAM solution using Session Manager, which is part of AWS System Manager. We also integrated Session Manager with key identity and access management systems in our enterprise. This approach allowed us to use current processes for managing access while providing a seamless user experience for those who access the EC2 instances.

Session Manager is provided at no additional cost for use with EC2 instances. Administrators can use Session Manager to grant and revoke access for both Windows and Linux instances from a single location through AWS Identity and Access Management (IAM) roles and policies. There is also no need for privilege account management and password rotation with Session Manager because Session Manager uses IAM roles and policies to check that the user is authorized for the target EC2 instance. Once authorized, Session Manager signs the user in with the system-generated OS account, ssm-user. To specify a different OS account, see enable run as support for Linux instances.

With Session Manager, EC2 instances do not require network level access, and therefore do not require you to add additional ingress rules to security groups for protocols such as SSH or RDP. Finally, Session Manager captures session histories and stores them in Amazon Simple Storage Service (Amazon S3) buckets and Amazon CloudWatch log groups.

Here are the steps we took to use Session Manager for PAM:

1. Complete prerequisites to enable Session Manager.
2. Integrate with enterprise Identity Management Service and Application Onboarding Service for access requests.
3. Integrate with Access Management Service for authentication and authorization.
   • 3a – Customize application launcher portal for selecting a target EC2 instance.
   • 3b – Set up SAML identity provider for authentication.
   • 3c – Set up IAM role for authorization.
4. Integrate with Amazon CloudWatch for auditing.

Step 1: Complete prerequisites to enable Session Manager

Our first step was to ensure that the OS on the target EC2 instance is supported and that the System Manager Agent (SSM Agent) is installed on the instance. For more information, see about SSM Agent in the AWS Systems Manager User Guide.

Because we use System Manager for our OS management, SSM Agent was already installed on all EC2 instances. Next, to enable access to SSM Agent, we had to create and attach an IAM instance profile with appropriate permissions to the instance. For the purposes of logging and AWS Key Management Service (AWS KMS) key session encryption (this is in addition to the TLS 1.2 encryption that AWS already provides by default), that instance profile must also have appropriate KMS, S3, and CloudWatch log group write permissions.

As with most AWS services, you can access Session Manager through the SDK, AWS Systems Manager console, or AWS Command Line Interface (AWS CLI), with the Session Manager plugin for the AWS CLI installed. Access through any of these methods requires valid AWS credentials. For AWS CLI, you can obtain AWS credentials through identity federation with Security Assertion Markup Language (SAML) in AWS Single Sign-On (SSO). Although Session Manager with AWS CLI offers more flexibility with session preferences by use of SSM documents, we decided that the integrated browser-based shell for Session Manager inside the AWS Systems Manager console is easier and quicker for users to access during a firecall. Unlike the CLI, the console interface doesn’t require the user to perform extra steps (for example, to install the AWS CLI and Session Manager plugin or for our team to have to create and maintain a bastion host with preinstalled CLI and plugins).

Step 2: Integrate with enterprise Identity Management Service and Application Onboarding Service for access requests

We decided to integrate our AWS PAM solution with our enterprise identity management systems. This integration allows us to use our existing centralized process around request, approval, and attestation of entitlements. Privileged access to EC2 instances from different tenant application groups are shown as different entitlements in our Identity Management Service and any user can request these entitlements.

At State Street, tenant applications are onboarded onto AWS using an internally developed Application Onboarding Service, which captures information about each onboarded tenant application, including those responsible for entitlement approvals and other cost management attributes for chargeback. It also uses AWS CloudFormation to provision different AWS resources, such as EC2 instances, required by the tenant application. For entitlements, the Application Onboarding Service uses AWS Managed Microsoft AD as its entitlement store where entitlements are represented as Active Directory (AD) groups. These entitlements are created during application onboarding and are later used to control access to different cloud capabilities, such as PAM.

[Note from AWS: AWS Service Catalog can be used as an alternative to the Application Onboarding Service mentioned here.]

Process flow

Figure 1: Process flow for requesting PAM access to a target EC2 instance

1. Applications are onboarded onto AWS through the Application Onboarding Service, which uses AWS CloudFormation to provision target EC2 instances
2. The Application Onboarding Service creates AD groups in AWS Directory Service for each logical subset of the application group’s EC2 instances.
3. AD groups are synchronized to State Street’s Identity Management Service through an integration job.
4. A user can request PAM access to a subset of the application group’s EC2 instances through the entitlement defined during onboarding.
5. Requests go through an approval process where the appropriate approvals are obtained before access is granted.
6. After approval, the Identity Management Service assigns the AD group to the user.

Step 3: Integrate with Access Management Service for authentication and authorization

State Street uses a commercial access management solution to perform single sign-on of different applications and systems with corporate Active Directory. This solution makes it possible for a user to sign in anywhere with their company credentials, including AWS. Our team also has a custom application launcher portal that integrates with the access management solution to display all the cloud capabilities, such as privileged access to EC2 instances and other cloud services we offer, available to the user based on the user’s AD group membership in the entitlement store.

Process flow

Figure 2: Accessing target EC2 instance after PAM EC2 approval is granted

1. The user opens the State Street internally developed application launcher portal.
2. The portal connects to State Street’s internal Access Management Service, a third-party commercial solution. The Access Management Service queries AWS Directory Service for the user’s AD group membership to determine the user’s access to different cloud capabilities.
3. For the cloud capability PAM, the application launcher portal uses AWS Config to query a list of available EC2 instances filtered by EC2 tag information embedded in the AD groups (described later in Step 3a). The user is then presented with a list of EC2 instances for PAM access. From this filtered list, the user selects the appropriate instance for the session.
4. The user is then redirected to the SAML endpoint as part of the connection process (described later in Step 3b).
5. As part of establishing a Session Manager session, IAM uses attribute-based access control (ABAC) to map attributes in SAML assertions, which contain EC2 instance tag information the user is authorized for, to the IAM role for PAM (described later in Step 3c).
6. The user is redirected to Session Manager where they are granted access to the target EC2 instance through Session Manager’s browser-based shell.

Step 3a: Customize the application launcher portal for selecting a target EC2 instance

Both the start-session CLI command and the Session Manager section of the AWS Systems Manager console require an EC2 instance ID to connect to an EC2 instance. Although the obvious solution is to use the EC2 DescribeInstances action to return an EC2 instance ID given tag filters, all EC2 APIs are subject to throttling restrictions. Due to the large size of State Street’s AWS environment and given that we already have AWS Config enabled, we selected a more robust approach and used the AWS Config SelectResourceConfig action.

Step 3b: Set up a SAML identity provider for authentication

We set up our company’s identity provider, Access Management Service, with AWS. The identity provider authenticates against our corporate AD before sending the SAML assertion to the AWS SAML Assertion Consumer Service (ACS) at https://signin.thinkwithwp.com/saml. For information about setting up a SAML identity provider, see creating IAM SAML identity providers in the AWS Identity and Access Management User Guide.

To have AWS redirect to Session Manager and launch an EC2 instance after it receives a SAML assertion, we set the SAML relay state for our identity provider to: https://console.thinkwithwp.com/systems-manager/session-manager/<instance id>?region=<region name>. For information about how to set the relay state for AWS Single Sign-on, see set relay state in the AWS Single Sign-On User Guide.

Step 3c: Set up an IAM role for authorization

It’s common to pass an IAM role name as part of the SAML assertion. However, this approach, using role-based access control (RBAC), would require us to create an IAM role for every target EC2 group. We decided that ABAC, which is available when using IAM, was a better approach. With ABAC, only one IAM role is required for all target EC2 groups. IAM permissions are enforced dynamically based on values passed in the SAML assertion.

Here is an example of an IAM policy for the IAM role for Session Manager that supports ABAC and conforms to the security principle of least privilege. The first action allows the user, which is a principal in IAM, to establish sessions through Session Manager to EC2 instances and managed instances, which are resources in IAM. Note the condition which allows access, only if the resources have resource tags that match the SAML attribute values in the user’s SAML assertions. The second action ensures that the user can only terminate their own sessions in Session Manager.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SSMStartSession",
            "Effect": "Allow",
            "Action": "ssm:StartSession",
            "Resource": [
                "arn:aws:ec2:*:*:instance/*",
                "arn:aws:ssm:*:*:managed-instance/*"
            ],
            "Condition": {
                "StringEquals": {
                    "ssm:resourceTag/APP_GROUP_NAME": "${aws:PrincipalTag/AppGroupName}",
                    "ssm:resourceTag/RELEASE_ID": "${aws:PrincipalTag/ReleaseId}",
                    "ssm:resourceTag/APP_ID ": "${aws:PrincipalTag/AppId}"
                }
            }
        },
        {
            "Sid": "SSMTerminateSession",
            "Effect": "Allow",
            "Action": "ssm:TerminateSession",
            "Resource": "arn:aws:ssm:*:*:session/*",
            "Condition": {
                "StringLike": {
                    "ssm:resourceTag/aws:ssmmessages:session-id": [
                        "${aws:userid}"
                    ]
                }
            }
        }
    ]
}

The scope of the IAM policy can be further restricted to specific account IDs, AWS Regions, or partitions. For more information, see quickstart default IAM policies for Session Manager in the AWS Systems Manager User Guide.

Using the IAM policy example, if a SAML assertion is sent to AWS with the following SAML attribute values:

<saml:Attribute Name="https://thinkwithwp.com/SAML/Attributes/PrincipalTag:AppGroupName"
    NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">webserver</saml:AttributeValue>
</saml:Attribute>

<saml:Attribute Name="https://thinkwithwp.com/SAML/Attributes/PrincipalTag:AppId"
    NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">ABC</saml:AttributeValue>
</saml:Attribute>

<saml:Attribute Name="https://thinkwithwp.com/SAML/Attributes/PrincipalTag:ReleaseId"
    NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">1.0</saml:AttributeValue>
</saml:Attribute>


<saml:Attribute Name="https://thinkwithwp.com/SAML/Attributes/Role"
    NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">arn:aws:iam::<ACCOUNT ID>:role/<IAM Role Name for PAM>,arn:aws:iam::<ACCOUNT ID>::saml-provider/<IAM Provider Name></saml:AttributeValue>
</saml:Attribute>

Based on the conditions in the IAM policy, Session Manager will only allow the user to connect to an EC2 instance with the matching tag values of:

• APP_GROUP_NAME = webserver
• APP_ID = ABC
• RELEASE_ID = 1.0

Access to other EC2 instances without those tags and tag values will be denied.

With the use of ABAC, different groups of EC2 instances using dynamic principal tag values can access Session Manager with only one IAM role. The Role SAML attribute supports multiple values in a single SAML assertion, which will be useful when we extend this approach to multiple AWS accounts and to accessing other AWS services. However, other SAML attributes for the principal tags don’t support multiple values in a single SAML assertion because the IAM policy condition uses the StringEquals operators to match on a single String value.

To support multiple groups of tags for a user at a given time, we suffix each PrincipalTag with a number for each group of tags. In the IAM policy, we have multiple policy statements, each matching one group of tags. For example, if the user is authorized for two AD groups (that is, two sets of EC2 tag values), the user would also have in the SAML assertion: PrincipalTag:AppGroupName2, PrincipalTag:AppId2, and PrincipalTag:ReleaseId2. In the IAM policy, we would have n IAM policy statements, where n is the maximum number of AD groups a user can have at a given time. Each set of policy statements is similar to the preceding example, but for PrincipalTag:<tag name>[2..n].

Although date conditions are supported in an IAM policy, there is currently no way to use a dynamically generated value from a SAML assertion for the date condition. As a workaround, we used functionality in our identity provider platform to time bound the access during the generation of the SAML assertion by setting one of the SAML attribute values to an invalid value when time is outside the allowed window.

Step 4: Integrate with Amazon CloudWatch for auditing

Session Manager can store session histories in CloudWatch log groups and S3 buckets. CloudWatch already integrates nicely with our team’s current logging infrastructure. Because the target EC2 instance is writing the session history logs to CloudWatch, the IAM role associated with the target EC2 (rather than the user’s assumed IAM role for PAM) needs IAM permissions for CloudWatch log groups and log streams. Session Manager currently stores all log events in the same log group. However, each session is stored in a unique log stream. The user name and session ID are part of the log stream name.

Figure 3: Log group details for session-manager

Before ingesting the CloudWatch data into our logging infrastructure, we use the Systems Manager describe-session action to get the EC2 instance ID and its tag values from the session ID. With the EC2 tag values, which contain the tenant application ID for the session’s EC2 instance, resulting logs are then ingested into the respective dedicated area for the corresponding tenant application team in our logging infrastructure.

(As of Nov 18, 2020, Session Manager added support for streaming to CloudWatch logs, which includes EC2 target instance id and the session id. Therefore, it is no longer necessary to use the describe-session call to capture this data. Incremental logging events are published as the session progresses.)

Conclusion

Session Manager allows you to access EC2 instances without opening inbound ports using IAM roles and offers native logging integrations with Amazon S3 and Amazon CloudWatch. This blog post shows how we were able to integrate Session Manager with enterprise identity and access management systems to create a seamless experience for the user for privileged access management to EC2 instances during a firecall.