AWS Cloud Operations Blog

Copy existing AWS CloudTrail trails events to a AWS CloudTrail Lake event data store

AWS announced the general availability of AWS CloudTrail Lake on 5th Jan 2022, a managed audit and security lake that lets you aggregate, immutably store, and query activity logs for auditing, security investigation, and operational troubleshooting. Since launch, customers have adopted this feature, and it’s an integral part of customer operational and security operational processes.

Today, we’re excited to announce a new import feature which will allow customers to copy existing trail events to a CloudTrail Lake event data store which were recorded prior to event data store creation. This new feature allows customers to import events from an Amazon Simple Storage Service (Amazon S3) buckets corresponding to a trail during auditing or troubleshooting. This also improves the efficiency during auditing, as well as security or operational incident troubleshooting. You can utilize SQL query language from a single source to find the required information. You no longer need to maintain multiple trails or data processing platforms. This new feature supports copying logs from an S3 bucket that stores logs from across multiple accounts (from an organization trail) and multiple AWS Regions. Furthermore, you can import logs from individual accounts and single-region trails. This feature will also let you specify an import date range, so that you only import the subset of logs that are needed for long-term storage and analysis in Lake.

In this post, I’ll walk you through this import feature that allows you to copy existing trail events to a CloudTrail Lake event data store. If you’re using CloudTrail Lake for the first time, then check this post.

Importing CloudTrail events from Trails page in CloudTrail console

  1. Open the AWS Console and sign in with an account with administrative permissions to manage CloudTrail.
  2. Navigate to CloudTrail console. Choose Trails in the left navigation pane of the CloudTrail console.
  3. Choose the trail that you want to copy and select Copy events to Lake.
Select Trail and choose on Copy events to Lake

Figure 1. Select Trail and choose on Copy events to Lake.

  1. (Optional) Choose the time range for importing events.
Select trail and choose time range

Figure 2. Select trail and choose time range.

  1. Select Lake event data store to copy trail events.
Select Lake event data store

Figure 3. Select Lake event data store.

  1. Select the AWS Identity and Access Management (IAM) role to copy events. You can either create a new role or select an existing role with adequate permission. You can read more about permissions required to copy trail events.
Create IAM role or select existing role with required permission.

Figure 4. Create IAM role or select existing role with required permission.

  1. Select copy events.
Select copy events

Figure 5. Select copy events.

  1. Check the status of the copying job by viewing the event data store.
Check status of copy job

Figure 6. Check status of copy job.

Now we’ve seen how to copy events from an existing trail into an event data store. Alternatively, you can copy events from Amazon S3, from the CloudTrail lake event data store page.

Importing CloudTrail events from the event data store details page in the CloudTrail console

  1. Open the Console and sign in with an account with administrative permissions to manage CloudTrail.
  2. Navigate to the CloudTrail console. Choose Lake in the left navigation pane of the CloudTrail console, and then choose Event data stores.
  3. On the Event data stores page, choose Copy trail events.
  4. Select the trail that you want to copy in the Events source drop-down list.
  5. (Optional) Choose the time range for copying events.
Select trail and time range to copy events

Figure 7. Select trail and time range to copy events.

  1. Choose the event data store from the drop-down list to copy trail events.
Select event data store where events will be copied

Figure 8. Select event data store where events will be copied.

  1. For Permissions, create a new role to copy trail events, or select an existing role from the drop-down list.
Create a new role or select existing role to copy events

Figure 9. Create a new role or select existing role to copy events.

  1. Choose Copy events to start importing.
Select copy events

Figure 10. Select copy events.

  1. On the Event copy status page, review the status for copying.
Review copy status

Figure 11. Review copy status.

  1. Review the details of the copy status.
Review details of copy status

Figure 12. Review details of copy status.

  1. Copy status details will show the following details of the trail event copy:
    1. Event S3 location
    2. S3 prefix copied
    3. Copy status
    4. Copy error details like error message, error location and error type

After the event copy status is complete, run an SQL query in the event data store to analyze events.

To get started, see Working with CloudTrail Lake Import feature in the CloudTrail User Guide.

Conclusion

In the post, we’ve announced the new CloudTrail Lake import feature. We’ve shown you how to copy events from CloudTrail trails directly, or from event data stores. To get started, see Working with CloudTrail Lake in the CloudTrail User Guide. Also, refer CloudTrail Lake getting started blog which demonstrate setting CloudTrail Lake and performing few example queries. We’re excited to make this new feature available for you, and we can’t wait to see what great things you build with it.

About the author:

Yagya Vir Singh

Yagya Vir Singh is a Senior Technical Account Manager based in Nashville, Tennessee. He is passionate about AWS technologies and loves to help customers achieve their goals. Outside of the office, he loves to be with his friends and family and spend time outdoors.