How to Manage Resources across Multiple Environments in AWS

To build great apps, startups typically require multiple environments, such as production, development, QA, and staging environments. But it can be challenging to track and manage resources over multiple environments, especially as a business grows over time.

For example, as your startup grows your developers might be spinning up resources much faster than you anticipated, and you might find that you need greater visibility into which resources belong to which environments.

How many instances belong to QA? Who has access to stop or start production instances? How much does the development environment cost? You’ll likely need to find answers to these types of questions as your startup grows.

You can use Amazon Web Services (AWS) not only to set up multiple environments, but also to track, manage, and control access to your resources across those environments. Let’s take a look at some of the AWS features that you can take advantage of.

Tags

You can use tags to identify, track, and manage your AWS resources across multiple environments (and even across multiple AWS accounts). A tag is a label that consists of a key and a value. You can tag most AWS resources, and you can add up to 10 tags per resource.

Tag everything you can because there are big advantages to using tags. For example, you can use tags for the following tasks:

• Find and organize your resources
• Control access to your resources
• View detailed billing reports

In addition, tagging your resources lets you take greater advantage of the following AWS features.

Resource Groups

Resource Groups is a cool new feature in the AWS Management Console that enables you to create, view, and maintain a collection of resources that share common tags. You can also create your own views of different tagged environments. For example, you can view all resources that have a single tag, such as Environment=Prod, across all services that support tagging. You can also view all resources that share multiple tags, as shown in the following screenshot.

For details, see Working with Resource Groups.

Using Tags with IAM

Do you want to lock down services or resources tagged as Prod to a specific set of users? You can use tags in conjunction with AWS Identity and Access Management (AWS IAM) to control the access of users or groups. For example, you can control which IAM users or groups can start, stop, reboot, and terminate specific EC2 instances. In this way, you could give a developer access to control only the instances that are tagged with an Environment=Dev tag.

For a complete list of the services that support IAM resource tagging, see AWS Services That Support IAM.

Using Tags with Billing Reports

Do you want to know how much a specific environment is costing? You can use cost allocation tags to categorize and track your AWS costs. When you apply tags to your AWS resources (such as Amazon EC2 instances or Amazon S3 buckets), AWS generates a cost allocation report as a comma-separated value (CSV file) with your usage and costs aggregated by your tags. You can apply tags that represent business categories (such as cost centers, application names, or owners) to organize your costs across multiple services.

The cost allocation report includes all of your AWS costs for each billing period. The report includes both tagged and untagged resources, so you can clearly organize the charges for resources. For example, if you tag resources with an application name, you can track the total cost of a single application that runs on those resources.

For more information, see Setting Up Your Monthly Cost Allocation Report.

Maintaining Tags

Now that we know the benefits of tagging our resources, how do we make sure everything that can be tagged gets tagged? To do this, you can use some of the tagging features in the AWS Management Console, such as adding bulk tags to instances. The following screenshot shows this type of tag management.

However, this does require you to log into the console and manually update resources when new ones are added, which can be a bit cumbersome over time. Following are some other options for tagging your resources.

Creating Your Own Launch Scripts

Providing your team members with a script or set of scripts to create the environments they need is a great way to make sure that resources are tagged correctly. It also makes their lives easier. They get the environment they want quickly, and it’s set up the right way.

Because every AWS service has an API, there are numerous ways to build scripts to create your resources. This could be as simple as a creating a shell script by using the AWS Command Line Interface (AWS CLI), or by using one of the many SDKs that are available for the platform. For more information about the SDKs, see Start Developing with Amazon Web Services.

There are also countless scripts available on sites like GitHub, which you can clone as a starting place.

Using CloudFormation Templates

Templates, like scripts, are a way to provide your teams with an easy, reproducible way to create resources on AWS. You can create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation will provision and configure those resources for you.

Almost every resource that supports tagging also supports tagging in CloudFormation templates. You can create templates for different environments, or you can use condition statements to use a single template for multiple environments.

Using Auto Scaling

The Auto Scaling service provides you with a number of great features. You can scale out EC2 instances as demand for your application increases, scale back down when instances are unneeded, and replace instances that become unresponsive.

You can also use tags with your auto-scaled instances. Instances that are launched as part of Auto Scaling are automatically tagged upon launch, saving you from creating a mechanism to do this.

Using Configuration Management

Many of the common configuration management tools, such as Chef, Puppet, SaltStack, and Ansible, support tagging of your resources upon creation. They also typically provide conditional separation for multiple environments. If you’re using one of these tools, chances are that you could easily implement a method for tagging created resources.

Finding Untagged Resources

From time to time, you might want to audit your account to identify untagged resources or to verify that resources are tagged correctly. Check out the following options for getting set up to audit your resources.

Using the AWS Management Console

The AWS Management Console provides an easy way to view resources based on tags, especially EC2 instances. You can use tags you created as columns for most resource views. This allows you to spot resources that have been either tagged incorrectly or not at all, as shown in the following screenshot.

Create Your Own Scripts

As mentioned earlier, AWS provides an API for accessing services. You can use the same API to query services for a list of resources. You can use these listings to find resources that are not tagged or are incorrectly tagged.

Summary

You can use tags to track and manage resources across multiple environments and to control access to those resources. You have several options for implementing tags in AWS. For example, you can use the AWS Management Console, CloudFormation templates, or your own custom scripts. Regardless of the option that you choose, you’ll find that tags are an essential way to manage your resources.

Follow us: @AWSstartups