Iterating Over Your Objects with Amazon S3

There are a lot of hidden gems inside the AWS SDK for Java, and we’ll be highlighting as many as we can through this blog.

Today, we look at how to interact with paginated object and version listings from Amazon S3. Normally, when you list the contents of your Amazon S3 bucket, you’re responsible for understanding that Amazon S3 returns paginated results. This means that you get a page of results (not necessarily the entire result set), and then have to use a nextToken parameter to request the next page of results, and repeat this process until you’ve read the complete data set.

Fortunately, the AWS SDK for Java provides some utilities to automatically handle these paginated result sets for you. The S3Objects and S3Versions classes allow you to easily iterate over objects and object versions in your Amazon S3 buckets, without having to explicitly deal with pagination.

Using these iterators to traverse objects in your bucket is easy. Instead of calling s3.listObjects(...) directly, just use one of the static methods in S3Objects, such as withPrefix(...) to get back an iterable list. This allows you to easily traverse all the object summaries for the objects in your bucket, without ever having to explicitly deal with pagination.

AmazonS3Client s3 = new AmazonS3Client(myCredentials);
for ( S3ObjectSummary summary : S3Objects.withPrefix(s3, "my-bucket", "photos/") ) {
    System.out.printf("Object with key '%s'n", summary.getKey());
}

If you’ve enabled object versioning for your buckets, then you can use the S3Versions class in exactly the same way to iterate through all the object versions in your buckets.

AmazonS3Client s3 = new AmazonS3Client(myCredentials);
for ( S3VersionSummary summary : S3Versions.forPrefix(s3, "my-bucket", "photos/") ) {
    System.out.printf("Version '%s' of key '%s'n", 
                      summary.getVersionId(), summary.getKey());
}