Announcing the AWS CloudTrail Processing Library

We’re excited to announce a new extension to the AWS SDK for Java: The AWS CloudTrail Processing Library.

AWS CloudTrail delivers log files containing AWS API activity to a customer’s Amazon S3 bucket. The AWS CloudTrail Processing Library makes it easy to build applications that read and process those CloudTrail logs and incorporate their own business logic. For example, developers can filter events by event source or event type, or persist events into a database such as Amazon RDS or Amazon Redshift or any third-party data store.

The AWS CloudTrail Processing Library, or CPL, eliminates the need to write code that polls Amazon SQS queues, reads and parses queue messages, downloads CloudTrail log files, and parses and serializes events in the log file. Using CPL, developers can read and process CloudTrail log files in as few as 10 lines of code. CPL handles transient and enduring failures related to network timeouts and inaccessible resources in a resilient and fault tolerant manner. CPL is built to scale easily and can process an unlimited number of log files in parallel. If needed, any number of hosts can each run CPL, processing the same S3 bucket and same SQS queue in parallel.

Getting started with CPL is easy. After configuring your AWS credentials and SQS queue, you simply implement a callback method to be called for every event, and start the AWSCloudTrailProcessingExecutor.

// This file contains your AWS security credentials and the name
// of an Amazon SQS queue to poll for updates
String myPropertiesFileName = "myCPL.properties";

// An EventsProcessor is what processes each event from AWS CloudTrail
final AmazonSNSClient sns = new AmazonSNSClient();
EventsProcessor eventsProcessor = new EventsProcessor() {
    public void process(List<CloudTrailEvent> events) {
        for (CloudTrailEvent event : events) {
            CloudTrailEventData data = event.getEventData();
            if (data.getEventSource().equals("ec2.amazonaws.com") &&
                data.getEventName().equals("ModifyVpcAttribute")) {
                System.out.println("Processing event: " + data.getRequestId());
                sns.publish(myQueueArn, "{ " + 
                    "'requestId'= '" + data.getRequestId() + "'," + 
                    "'request'  = '" + data.getRequestParameters() + "'," + 
                    "'response' = '" + data.getResponseElements() + "'," +
                    "'source'   = '" + data.getEventSource() + "'," +
                    "'eventName'= '" + data.getEventName() + "'" +
                    "}");
            }
        }
    }
};

// Create AWSCloudTrailProcessingExecutor and start it
final AWSCloudTrailProcessingExecutor executor = 
            new AWSCloudTrailProcessingExecutor
                .Builder(eventsProcessor, myPropertiesFileName)
                .build();
executor.start();

The preceding example creates an implementation of EventsProcessor that processes each of our events. If the event was from a user modifying an Amazon EC2 VPC through the ModifyVPCAttribute operation, then this code publishes a message to an Amazon SNS topic, so that an operator can review this potentially large change to the account’s VPC configuration.

This example shows how easy it is to use the CPL to process your AWS CloudTrail events. You’ve seen how to create your own implementation of EventsProcessor to specify your own custom logic for acting on CloudTrail events. In addition to EventsProcessor, you can also control the behavior of AWSCloudTrailProcessingExecutor with these interfaces:

• EventFilter allows you to easily filter specific events that you want to process. For example, if you only want to process CloudTrail events in a specific region, or from a specific service, you can use a EventFilter to easily select those events.
• SourceFilters allow you to perform filtering using data specific to the source of the events. In this case, the SQSBasedSource contains additional information you can use for filtering, such as how many times a message has been delivered.
• ProgressReporters allow you to report back progress through your application so you can tell your users how far along in the processing your application is.
• ExceptionHandlers allow you to add custom error handling for any errors encountered during event processing.

You can find the full source for the AWS CloudTrail Processing Library in the aws-cloudtrail-processing-library project on GitHub, and you can easily pick up the CPL as a dependency in your Maven-based projects:

<dependency>
	<groupId>com.amazonaws</groupId>
	<artifactId>aws-cloudtrail-processing-library</artifactId>
	<version>1.0.0</version>
</dependency>

For more information, go to the CloudTrail FAQ and documentation.

How are you using AWS CloudTrail to track your AWS usage?