Amazon S3 TransferManager

One of the great APIs inside the AWS SDK for Java is a class called TransferManager that makes working with uploads and downloads from Amazon S3 easy and convenient.

TransferManager provides asynchronous management for uploads and downloads between your application and Amazon S3. You can easily check on the status of your transfers, add handlers to run code when a transfer completes, cancel transfers, and more.

But perhaps the best thing about TransferManager is how it hides the complexity of transferring files behind an extremely simple API. TransferManager is essentially two operations: upload and download. From there you just work with your upload and download objects to interact with your transfers. The following example shows how easy it is to create a TransferManager instance, upload a file, and print out its progress as a percent while it’s transferring.

// Each instance of TransferManager maintains its own thread pool
// where transfers are processed, so share an instance when possible
TransferManager tx = new TransferManager(credentials);

// The upload and download methods return immediately, while
// TransferManager processes the transfer in the background thread pool
Upload upload = tx.upload(bucketName, myFile.getName(), myFile);

// While the transfer is processing, you can work with the transfer object
while (upload.isDone() == false) {
    System.out.println(upload.getProgress().getPercentTransferred() + "%");
}

Behind this simple API, TransferManager is doing a lot of work for you. Depending on the size and data source for your upload, TransferManager adjusts the algorithm it uses to process your transfer, in order to get the best performance and reliability. Whenever possible, uploads are broken up into multiple pieces, so that several pieces can be sent in parallel to provide better throughput. In addition to higher throughput, this approach also enables more robust transfers, since an I/O error in any individual piece means the SDK only needs to retransmit the one affected piece, and not the entire transfer.

TransferManager includes several more advanced features, such as recursively downloading entire sections of S3 buckets, or the ability to clean up pieces of failed multipart uploads. One of the more commonly used options is the ability to attach a progress listener to your uploads and downloads, which can run custom code at different points in the transfer’s lifecycle. The following example demonstrates using a progress listener to periodically print out the transfer’s progress, and print a final message when the transfer completes.

TransferManager tx = new TransferManager(credentials);
Upload upload = tx.upload(bucketName, myFile.getName(), myFile);

// You can set a progress listener directly on a transfer, or you can pass one into
// the upload object to have it attached to the transfer as soon as it starts
upload.setProgressListener(new ProgressListener() {
    // This method is called periodically as your transfer progresses
    public void progressChanged(ProgressEvent progressEvent) {
        System.out.println(upload.getProgress().getPercentTransferred() + "%");

        if (progressEvent.getEventCode() == ProgressEvent.COMPLETED_EVENT_CODE) {
            System.out.println("Upload complete!!!");
        }
    }
};

// waitForCompletion blocks the current thread until the transfer completes
// and will throw an AmazonClientException or AmazonServiceException if
// anything went wrong.
upload.waitForCompletion();

For a complete example of using Amazon S3 TransferManager and progress listeners, see the AmazonS3TransferManager sample that ships with the SDK for Java.

Are you using TransferManager in any of your projects yet? What custom code do you run in your progress listeners? Let us know in the comments!