AWS Cloud Operations Blog

Using AWS OpsWorks for Chef Automate in a federated environment

Many large enterprises operate on a federated model. That is, they are separated into different business units or organizations, with different goals, procedures, and skill sets. These enterprises typically use a system to manage their infrastructure configuration and changes.

You might ask, “Can we apply a federated model to configuration management? If so, what are the pros and cons and how does AWS OpsWorks for Chef Automate fit into that model?”

In this blog post we discuss how to approach configuration management in a federated enterprise.  The main focus is on how you can leverage AWS OpsWorks for Chef Automate to achieve this goal.

Single Chef server to serve all parts of your organization

Approaching configuration management in a non-federated way involves using a single Chef server across many different business units.

But, I hear you say, “Different business units have their own Chef nodes, cookbooks, and environments. How do we separate those?”

The way you can achieve this with Chef is to use the concept of Chef Organizations, which is a core part of the Chef Server software.

Chef Organizations allows you to separate the following items among different business units, but they still reside on a single Chef server:

  • Permissions
  • Nodes
  • Roles
  • Environments
  • Cookbooks
  • Data bags

The main benefits of this approach is that each business unit can independently manage its own nodes, access permissions, and can perform maintenance/updates on its infrastructure on its own schedule. Having this type of isolation allows the business units/organizations to have access to their own unique environments, roles, data bags, and cookbooks. Additionally, by hosting all of this on a single server you reduce the amount of effort required to set up and maintain a separate Chef server for each business unit.

Single Chef Server Model

Single Chef server per business unit

When we speak about a model for a true federated approach to configuration management, we imply that each of the company’s business units/organizations has its own separate Chef server. This brings us the same benefits that we get when we use Chef Organizations, but with even greater isolation between the business units. This brings more benefits when in the cloud.

“What are these new benefits you speak of?”

Well, it is very common to have different business units, each with its own respective AWS Account, for all of their resources. This approach is used for billing. It gives us a clear separation of resource usage per business unit. If each business unit uses a Chef server in a centralized AWS account, billing and management of that resource can get out of hand. Another major benefit, which is not strictly cloud related, is that the maintenance of the Chef server and the potential downtime it experiences, does not cause disruptions across all organizations and their infrastructure. By having a Chef server for each business unit you can mitigate the issue of having a single point of failure. If you rely on a single Chef server throughout your enterprise, that server’s failure would cause potential down time across all business units. By using multiple Chef servers you reduce the blast radius of potential Chef server down time.

Federated model/Multi Chef Server

Additionally, the concept of Chef Organizations can be fully utilized even in a federated model. For example, if a certain business unit wants to separate even further, it can split its own Chef Automate server into organizations. The nodes from all organizations will be visible on the Chef Automate dashboard because you can filter nodes based on Chef Organizations.

“But what happens to our cookbooks? Does each business unit need to write its own?” To answer these questions, let’s take a look at Centralized Cookbook repositories.

Centralized Cookbook repositories

A great way to make all of your configurations as consistent as possible is to have a companywide Cookbook repository. This means that teams/business units can use cookbooks created by other parts of the company to configure their infrastructure – on their respective Chef Servers. This, in turn, reduces the effort required to start using Chef as a configuration management system in a business unit because most of the required code base is already present. So, for example, let’s say that business unit A writes its own cookbook for setting up an application, and business unit B also uses the same application in their own environment. Business unit B does not have to write its own cookbook from scratch, but can use the cookbooks already created by business unit A. The main obstacle to having a shared cookbook system is properly writing cookbooks in the first place. Each cookbook must be as dynamic as possible. When you write a cookbook, leverage attributes, templates, and data-bags as much as possible so you can be flexible and help other teams use the same cookbook for,  their use cases.

“So what kind of Cookbook repos can we use?”

Git repo

The simplest approach to centrally managing your cookbooks is to use  Git repositories. Yes, simple as that. You can use the Berkshelf tool to manage the cookbooks, but instead of pulling them from the Chef supermarket, it pulls from the Git repo. With Berkshelf, you can specify a certain cookbook version to be pulled. In addition, because Berkshelf supports specifying a repository branch, you can even select a specific branch of your cookbook repo, for example, a certain testing branch to be used in a Dev environment.

Berksfile example:

cookbook "buk-cookbook", "~>; 0.9.3", git: "https://github.com/example/buk-cookbook.git"
cookbook "jinar-cookbook", "~>; 0.2.4", git: "https://github.com/example/jinar-cookbook.git", branch: "my-test-branch"

As mentioned before – you can have centralized Cookbook Git repositories across the entire company. Then each team could create its own repositories and push cookbooks – which could, in turn, be used by everyone in the company.

Common GIT repositories for cookbooks

“But–hey – at the beginning of this blog post you mentioned that one of the benefits of using a single Chef server is not needing to configure and maintain each server separately. Doesn’t having too many Chef servers give us a lot of overhead?”

Enter – OpsWorks for Chef Automate

How does OpsWorks for Chef automate fit into all of this?

AWS OpsWorks for Chef Automate (OWCA) provides a fully managed Chef server and suite of automation tools, which compose Chef Automate. OpsWorks for Chef Automate offers you a consistent Chef server configuration throughout all deployed servers. All servers are deployed and bootstrapped by OpsWorks, so this means that each new server will be configured in the same way. Additionally, OpsWorks takes care of the Chef server maintenance in the form of server patching, minor Chef updates, and backups. That means that you do not have to spend time and effort in order to set-up and configure a Chef Automate server. You can just let OpsWorks handle that for you, and have your business units focus all their energy on writing or using Chef cookbooks in their environments.

The fact that OpsWorks handles patching means that you don’t need to connect to the Chef server using SSH. You only  need to use knife to manage your Chef server software. With backup and restore in place, you can easily revert back to a working version of your server if something goes wrong.

A 10 000 foot view of setting up OWCA for a business unit would look something like this:

  • Launch an OWCA server in the desired Region with the desired instance size.
  • Take the starter kit provided with the Chef Automate server to be used on a Chef Workstation.
  • Once the server is fully running, dministrators can use the Chef Workstation to create all the required resources:
    • Environments
    • Roles
  • Set Berkshelf to get the needed Cookbooks from a common repository, and have them installed on the Chef Automate server.
  • Finally, bootstrap nodes in the appropriate roles. The usage of AWS API can be applied here, as OpsWorks can handle node bootstrapping for you.

Simple as that! You don’t need to manually configure Chef servers. You  just  launch an OpsWorks for Chef Automate server, and the rest is just Cookbook and node management.

Summary

Here we have discussed how to implement a federated model for configuration management using Chef and OpsWorks. We described how simple it is to set up and maintain a Chef server with OpsWorks. Many of our customers are actually taking this direction when using OpsWorks for Chef Automate because it allows them to separate their teams but still maintain a common set of configuration rules across the entire enterprise. With the assistance of the AWS API to bootstrap nodes this allows you to spend less time worrying about setting up the configuration management infrastructure and more on creating and using Cookbooks and any other Chef resources you need.

About the Author


Darko Meszaros is a Cloud Support Engineer who supports customers that use various AWS automation tools, such as AWS OpsWorks, AWS CodeDeploy, and AWS CloudFormation. He is a subject matter expert for OpsWorks and CodeDeploy. Outside of work, he loves collecting video games and old computers.

 

 

The screenshots in this post are Copyright Chef Software Inc., and are available at https://github.com/chef/chef-web-docs under the terms of the CC-BY-3.0 license, available here https://creativecommons.org/licenses/by/3.0/legalcode.