Analyze Active Directory Event logs using Amazon OpenSearch

Today, security and operation teams have to rely on security events from multiple Active Directory (AD) domain controllers for security analysis. Some key insights from the security analysis include changes to Windows Firewall, users or groups, Windows Registry, and failed login attempts etc. Centralizing these events from domain controllers and domain joined servers and computers will help security and operation teams analyze and visualize security data to rapidly identify the root cause of potential security issues.

In this blog post, we will discuss a high-level design on how to stream event logs from AD domain controllers and domain joined Amazon Elastic Compute Cloud (Amazon EC2) instances using Amazon CloudWatch Logs and AWS Lambda into Amazon OpenSearch. After consolidating the logs, we will create custom dashboards using Kibana. For additional security, we can integrate AWS Managed Microsoft AD with AWS Single Sign-On (AWS SSO) to authenticate and authorize AD users/groups to Kibana dashboards.

Many enterprise customers who want to use the OpenSearch dashboard capabilities find it challenging to secure access to the dashboard. The OpenSearch dashboard users have direct access to data stored in Amazon OpenSearch – so it’s important that only authorized users have access to Kibana.

AWS SSO simplifies managing SSO access to AWS accounts and business applications, and it’s the central location where you can create or connect your workforce identities in AWS. You can control SSO access and user permissions across all your AWS accounts in AWS Organizations.

AWS Managed Microsoft AD lets you run Microsoft AD as a managed service powered by Windows Server. When you select and launch this directory type, it creates a highly available pair of domain controllers connected to your Amazon Virtual Private Cloud (VPC). The domain controllers run in different Availability Zones within the AWS Region of your choice.

Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. Amazon OpenSearch Service supports OpenSearch and legacy Elasticsearch OSS. When you create a cluster, you have the option of which search engine to use.

High-level Architecture

Figure 1: High-level architecture pattern for analyzing AD logs.

Create AWS Managed Microsoft AD (If you do not have one)

1. In the AWS Directory Service console navigation pane, choose Directories and then choose Set up directory.
2. On the Select directory type page, choose AWS Managed Microsoft AD, and then choose Next.
3. On the Enter directory information page, provide the information such as Directory Edition, Directory DNS name, Directory NetBIOS name, Password, etc.
4. On the Choose VPC and subnets page, provide the information about the VPC and Subnets that you want to associate your directory to.
5. On the Review & create page, review the directory information and make any necessary changes. When the information is correct, choose Create directory. Creating the directory takes 20 to 40 minutes. Once created, the Status value changes to Active.

Figure 2: Directory Service – AWS Managed Microsoft AD

Enable Managed AD Log Forwarding 

We can use either AWS Directory Service console or APIs to forward the domain controller’s security event logs to Amazon CloudWatch Logs. Once log forwarding is configured, the security event logs will be available in CloudWatch Logs, which can be streamed to Amazon OpenSearch service for monitoring and analysis.

1. In the AWS Directory Service console navigation pane, choose Directories.
2. Choose the directory ID of the AWS Managed Microsoft AD directory that you want to enable the log forwarding for.
3. On the Directory details page, scroll down to the Log Forwarding section and choose enable.
4. On the Enable log forwarding to CloudWatch dialog, choose either of the following options:
   1. Select Create a new CloudWatch log group, under CloudWatch Log group name, specify a name that you can refer to in CloudWatch Logs.
   2. Select Choose an existing CloudWatch log group, and under Existing CloudWatch log groups, select a log group from the menu.
5. Review the pricing information and link, and then choose Enable.

Figure 3: Enable Log Forwarding in AWS Managed Microsoft AD

Setup Domain Instance Log Forwarding

Use CloudWatch Logs agent to push security event logs from on-premises Windows Server or Amazon EC2 to Amazon CloudWatch Logs. See steps below for installing and configuring CloudWatch agent.

1. Install CloudWatch agent on the Windows Server instances. For step-by-step guidance read Install Amazon CloudWatch agent
2. Once installed, run the agent as an Administrator and finish the configuration wizard. This will create a config.json file. For a step-by-step guide on how to configure, read Amazon CloudWatch agent configuration.
3. Start Amazon CloudWatch agent to fetch the logs.
4. Open Amazon CloudWatch console and verify the logs are streamed to the log group.

Figure 4: Amazon CloudWatch Log stream events

Create Amazon OpenSearch Domain

Create Amazon OpenSearch domain for collecting the CloudWatch logs. For step-by-step guidance on creating OpenSearch domain, refer to Create an OpenSearch Service domain.

We have created an OpenSearch domain with 3 data nodes and no dedicated master node. It’s a best practice to have dedicated master nodes for production cluster. You can learn more about best practices for Amazon OpenSearch domain here – Best practices for Amazon OpenSearch Service

Figure 5: Amazon OpenSearch configuration

Configure AWS SSO for OpenSearch Dashboard 

We are using AWS SSO managed users/groups to perform authentication for OpenSearch dashboard. AWS SSO uses standard identity federation via Security Assertion Markup Language (SAML) similar to Microsoft ADFS or Ping Federation. AWS SSO integrates with AWS Managed Microsoft AD or Active Directory hosted on premises or an Amazon EC2 instance through AWS Active Directory Connector.

To configure AWS SSO for OpenSearch dashboard, follow the steps from these two blog posts :

How to enable secure access to Kibana using AWS Single Sign-On

How to Integrate AWS SSO with AWS Managed Elasticsearch Kibana Instance

Stream AD logs to Amazon OpenSearch Domain

In our setup, we have already streamed the domain controller security event logs to Amazon CloudWatch Logs. Now, you can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription.

Follow the step-by-step AWS guide to configure the CloudWatch log subscription. CloudWatch log subscription creates a AWS Lambda function and you can write your own code to transform the security event logs before it reaches the OpenSearch domain.

Figure 6: Subscription to Amazon CloudWatch log groups

Figure 7: Subscription filter added for AD events

Create Visualization in Amazon OpenSearch

Now that data is flowing through the system, we can create simple visualizations to analyze this data in real-time.

Amazon OpenSearch visualizations are based on Amazon OpenSearch queries and the data in your OpenSearch indexes. You can create charts that show trends, spikes, and dips by using a series of Amazon OpenSearch aggregations to extract and process data.

To create visualizations,

1. Open the OpenSearch dashboard and use the visualize function.
2. Click on Create Visualization and select the Visualization Type that suits your use case. For this blog post, we are using pie chart, vertical bar and data table visualization types.
3. Select the data source for the visualization, you can choose either a saved search (OpenSearch query) or an index pattern.
4. After choosing a visualization format and data source, you will have to configure the visualization parameters such as data, metrics & axes and panel. The visualization parameters vary with each visualization type.

We have created a dashboard with three visualizations showing Audit Status, System Access Types, and Log Sources for the security event logs.

Figure 8: Amazon OpenSerach dashboard with AD log data

Furthermore, you can use the Discover option in the OpenSearch dashboard to explore your data interactively. You can access every document in every index that matches the selected index pattern.

Figure 9: Amazon OpenSearch data

Conclusion

In this blog post, we streamed the events from AWS Managed AD, domain joined Amazon EC2 instances to Amazon OpenSearch Domain, and created dashboards from the centralized log data to gain operational insights. This approach can be used for a self-managed Microsoft AD on AWS as well. Since domain controllers for self-managed Microsoft AD is installed on Amazon EC2 instances, you can follow the same integration pattern to stream the events.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your modernization journey today.