Simplify analysis of AWS CloudTrail data leveraging Amazon CloudWatch machine learning and advanced capabilities

AWS CloudTrail tracks user and API activities across AWS environments for governance and auditing purposes and allows customers to centralize a record of these activities. Customers have the option to send AWS CloudTrail logs to Amazon CloudWatch that simplifies and streamlines the analysis and monitoring of AWS CloudTrail recorded activities.

Amazon CloudWatch anomaly detection allows customers to surface deviance from normal behavior with minimal effort. It leverages machine learning algorithms to automatically identify unusual patterns or deviations in API activities. This coupled with AWS CloudTrail events provides visibility into potential security concerns and suspicious behavior within your AWS infrastructure. Amazon CloudWatch Logs provide the ability to query CloudTrail logs. In addition, customers have the option to leverage Amazon CloudWatch Metric Insights to perform SQL like query on the custom metrics created from CloudTrail logs.

In this blog post, we will explore how you can leverage advanced capabilities of Amazon CloudWatch to automate monitoring of AWS CloudTrail recorded events and gain valuable insights from your CloudTrail logs.

Solution Walkthrough

In this solution, we demonstrate Amazon CloudWatch’s machine learning and advanced capabilities by following a scenario to monitor and analyze access pattern for successfully retrieving Amazon Simple Storage Service (S3) objects using presigned URLs across AWS organization. S3 presigned URLs allow users to share or upload objects to S3 buckets through a simple URL. The capabilities of a presigned URL are limited by the permissions of the user who created it. Users can set an expiration time for the URL or can expire the URL by revoking the permissions of the user that created it.

For our sample scenario, we want only a certain set of IAM users to be able grant GetObject access to S3 objects using presigned URL and monitor our organization for any deviation from this behavior. To do so, we leverage Amazon CloudWatch to create a custom metric on top of AWS CloudTrail logs to surface information on successful GetObject calls to S3 objects using presigned URL. We detect anomalies in the expected values of this metric using CloudWatch anomaly detection. After we are alarmed of a potential anomaly, we leverage CloudWatch Metric Insights and Logs Insights to analyze if the access pattern behavior pattern is expected and take required action.

Pre-requisites

1. Access to or ability to create an organizational Trail
2. An existing S3 bucket with objects that you want to monitor

Step 1: Configure organizational Trail

For this exercise, you can use an existing CloudTrail organizational trail that is delivering to CloudWatch. The steps below walk you through to create a CloudTrail organizational trail and enable Data Events for your S3 bucket. Please note the first delivery of each management event for an account to your S3 bucket is free and data events are charged as per data events delivered.

1. Navigate to CloudTrail console. From left hand pane, choose Trails and then from right hand pane select Create Trail.
2.  On Choose trail attributes page, provide a trail name such as advanced-cloudtrail-events. Check the box for Enable for all accounts in my organization.
3.  For simplicity, we have not enabled Log file SSE-KMS encryption.
4.  Underneath CloudWatch Logs, check CloudWatch Logs as Enabled.
   1. Choose New to create a new log group. Keep Log group name as it is. Note down the log group name.
   2. Choose New to create a new IAM role for permissions to send logs to CloudWatch Log. Provide Role name such as advanced-cloudtrail-events-role
5. On Choose log events:
   1. Keep Management events box checked
   2. Check box for Data events
   3. Under Data events, for Data event type select S3. Under Log selector template select Custom.
   4. Under Advanced event selectors, for Field choose resources.ARN, under Operator choose starts with and under Value provide the ARN of S3 bucket you want to monitor as per pre-requisite 2. Select Next.
6. On Review and Create page, select Create Trail.

Step 2: Create custom Metric from CloudTrail logs

We now create a custom metric that will log successful access to an S3 object using presigned URL created by an IAM user. AWS CloudTrail logs contain fields that help understand details such what credentials were used to make a request, what IP address request was made from, what action was taken etc. You can create metric filters to match terms in your log events and convert log data into metrics. S3 presigned URLs successful attempts are recorded by CloudTrail events under additionalEventData event field with AuthenticationMethod as QueryString. To capture S3 presigned URLs successful attempts, please follow steps below:

1. Navigate to the CloudWatch console. From left hand panel, choose Logs, and then choose Log groups.
2. From right hand panel, underneath Log groups filter by log group name your noted in Step 1.4a and choose the log group.
3. Choose Metric Filters tab and select Create Metric Filter.
4. On Define pattern page, under Filter Pattern, provide pattern as below and Select Next
   { ($.eventName = GetObject) && ($.additionalEventData.AuthenticationMethod=QueryString) }
5. On Assign Metric page:
   1. Under Create filter name, enter Filter name such as presigned-URL-filter
   2. Under Metric details, for Metric namespace, make sure that Create new is selected and enter a name for the CloudWatch namespace such as advanced-cloudtrail-event where the metric will be published.
   3. Enter Metric name such as presigned-URL. Enter Metric value as 1 and Unit as Count. Select Next.
   4. Leave Dimensions as it is
      Figure 1: Create Metric
6. On Review and create page, choose Create metric filter.

You will be automatically navigated to content page and should now notice custom metric created under Metric filters pane which will track successful GetObject calls to S3 objects using preSigned URLs.

Figure 2: Created Amazon CloudWatch custom metric

Enable anomaly detection on custom metric

When you enable anomaly detection for a metric, CloudWatch applies statistical and machine learning algorithms. These algorithms continuously analyze metrics of systems and applications, determine normal baselines, and surface anomalies with minimal user intervention.

To enable Anomaly detection on the custom metric you just created, in the presigned-URL-filter, under Metrics choose presigned-URL. Select the advance-cloudtrail-event namespace that you previously created and drill down into ‘Metrics with no dimensions’.

Navigate to Graphed metrics tab and select anomaly detection icon. Make sure the Statistic set to Sum.
Figure 3: Enable Amazon CloudWatch Anomaly Detection

Anomaly Detection (AD) gets enabled immediately. A model is created based on the metric data points for a two week period. AD can also be enabled even if there is no data available for two week period. If you notice an exclamation point next to the (expected) chart legend, that means the model does not have enough data points to create a band.

Figure 4: Expected value range bands for custom metric

Follow steps mentioned in create a CloudWatch alarm based on anomaly detection documentation to alert you when metric behavior deviates from expected behavior.

Now that we have learnt how to automatically monitor our CloudTrail activity using custom metric and anomaly detection, let us understand how CloudWatch capabilities can help in analyzing CloudTrail data.

Step3: Analyze CloudTrail activity using Metric insight

Amazon CloudWatch allows you to extract fields from CloudTrail logs and present them as a metric characteristic using Dimensions. Please note that each new dimension is a new custom metric and is priced as per standard Amazon CloudWatch pricing.

Let’s create a new metric that counts successful S3 object access using presigned URL and also logs user name of the user that created the presigned URL. This will help us check if presigned URL is generated by approved users. If we detect a new/unapproved user, we will revoke the user S3 access which will in turn expire presigned URL. To do so, create a new metric following step 2.1 and 2.6 with below changes:

1. In step 2.4, modify filter pattern below:
   { ($.eventName = GetObject) && ($.additionalEventData.AuthenticationMethod=QueryString) && ($.userIdentity.userName=*) }
2. In step 2.5a modify Metric Name to presigned-URL-userName
3. In step 2.5d, on Dimensions panel enter Dimension Name as userName and Dimension Value as $.userIdentity.userName

  Figure 5: Add dimension to custom metric

Now once an anomaly is detected and an alert is sent out, we can leverage Metric Insights to investigate the access pattern. To do so, follow the steps below:

1. Navigate to the CloudWatch console. From left hand panel, choose Metrics, and then choose All Metrics.
2. On the right panel, choose Query tab and then choose Editor. Enter the below query to visualize IAM users that generated presigned URLs which account for successful calls to S3 bucket objects:
   SELECT SUM("presigned-URL-userName") FROM "advanced-cloudtrail-event" GROUP BY userName

  Figure 6: Successful S3 object accesses using presigned URL per IAM user

Note the legend at the bottom of the graph and you should notice a new IAM user unauthorizedUser has caused the spike. To investigate what AWS account this user is in, please follow the below steps:

1. Navigate to the CloudWatch console. From left hand panel, choose Logs, and then choose Log Insights.
2. From right panel, underneath Logs Insights choose log group created in step 1.4a and enter the query below
   filter userIdentity.userName ="unauthorizedUser"
stats count(*) by userIdentity.accountId

  Figure 7: Account ID for unauthorized user

You now have the accountID(s) where user exists and you can check if this is expected behavior or choose to the revoke user access.

Conclusion

In this blog post, we showed you how you can leverage advanced Amazon CloudWatch capabilities to simplify analysis of CloudTrail events across your AWS infrastructure through continuous automated monitoring and analysis of Amazon CloudTrail logs. You can now create custom metric, apply anomaly detection and query the metric data using metric insights on CloudTrail log data. To learn about each of these feature please visit CloudWatch log Insights, Anomaly detection, Metrics insights documentation and to know more about Amazon CloudWatch visit Getting Started with Amazon CloudWatch documentation.

About the authors