Amazon S3 Lifecycle Management

Amazon Simple Storage Service (S3) provides a simple method to control the lifecycle of your S3 objects. In this post, we examine how you can easily set up rules to delete or archive old data in S3 using the AWS SDK for .NET.

Lifecycle Rules

Lifecycle configurations are associated with a bucket. A lifecycle configuration consists of a number of rules, with each rule specifying the objects it acts on and the actions to take. Rules specify which objects they act on by defining a prefix. A rule can archive an object to Amazon Glacier, delete an object, or both. The action associated with a rule specifies a time constraint on it, acting on the objects that are either older than a specific number of days, or after a particular date. A rule also has a Status, which can be set to Enabled or Disabled. If you haven’t set this field, the rule will be disabled by default.

For instance, it’s possible to configure a rule that all objects with the prefix “logs/” must be archived to Glacier after one month. Here is a rule that does just that:

var rule1 = new LifecycleRule
{
    Prefix = "logs/", 
    Transition = new LifecycleTransition
    {
        Days = 30,
        StorageClass = S3StorageClass.Glacier
    },
    Status = LifecycleRuleStatus.Enabled
};

Rules can also be configured for a specific date. The following rule is configured to delete all objects with the prefix “june/” on the 1st of August, 2014.

var rule2 = new LifecycleRule
{
    Prefix = "june/", 
    Expiration = new LifecycleRuleExpiration
    {
        Date = new DateTime(2014, 08, 01)
    },
    Status = LifecycleRuleStatus.Enabled
};

Finally, a rule can contain both a transition and an expiration command. The following rule transitions objects to Glacier after 2 months and deletes objects after 1 year. This sample also configures a disabled rule.

var rule3 = new LifecycleRule
{
    Prefix = "user-data/",
    Status = LifecycleRuleStatus.Disabled,
    Transition = new LifecycleTransition
    {
        Days = 60,
        StorageClass = S3StorageClass.Glacier
    },
    Expiration = new LifecycleRuleExpiration
    {
        Days = 365
    },
    Status = LifecycleRuleStatus.Disabled
};

Lifecycle Configuration

A lifecycle configuration is simply a list of rules. In the following example, we construct a lifecycle configuration that consists of the rules we created earlier, and then this configuration is applied to our test bucket.

S3Client.PutLifecycleConfiguration(new PutLifecycleConfigurationRequest
{
    BucketName = "sample-bucket",
    Configuration = new LifecycleConfiguration
    {
        Rules = new List { rule1, rule2, rule3 }
    }
});

When dealing with configurations, we must configure all rules on a bucket. This means that if you wish to modify or add new rules, you must first retrieve the current configuration, modify it, and then apply it to the bucket. The following sample shows how we can enable all disabled rules and remove a specific rule.

// Retrieve current configuration
var configuration = S3Client.GetLifecycleConfiguration(
new GetLifecycleConfigurationRequest
{
    BucketName = "sample-bucket"
}).Configuration;

// Remove rule with prefix 'june/'
configuration.Rules.Remove(configuration.Rules.Find(r => r.Prefix == "june/"));
// Enable all disabled rules
foreach (var rule in configuration.Rules)
    if (rule.Status == LifecycleRuleStatus.Disabled)
        rule.Status = LifecycleRuleStatus.Enabled;

// Save the updated configuration
S3Client.PutLifecycleConfiguration(new PutLifecycleConfigurationRequest
{
    BucketName = "sample-bucket",
    Configuration = configuration
});

Finally, if you want to turn off all lifecycle rules for a bucket, you must either disable all rules (by setting Status = LifecycleRuleStatus.Disabled) or call the DeleteLifecycleConfiguration method, as follows.

// Remove a bucket's lifecycle configuration
S3Client.DeleteLifecycleConfiguration(new DeleteLifecycleConfigurationRequest
{
    BucketName = "sample-bucket"
});

Summary

In this blog post, we’ve shown how simple it is to configure the lifecycle of your S3 objects. For more information on this topic, see S3 Object Lifecycle Management.