AWS Cloud Operations Blog
GoDaddy’s journey to the cloud and their Public Cloud Portal
Get to the Cloud. Make awesome happen!
This blog post explores GoDaddy’s journey to the cloud and the GoDaddy Public Cloud Portal, an application used by engineering teams to onboard to AWS. GoDaddy started their journey to the cloud in early 2018 when they announced their partnership with AWS. This post will outline the project that enables GoDaddy engineering teams to easily and quickly navigate through the complete on-boarding workflow, speeding their journey to the AWS cloud.
The main purpose of the GoDaddy Public Cloud Portal is to enable the overall experience for GoDaddy teams to ‘Build better products faster’. The portal delivers a seamless one stop shop for GoDaddy developers to learn, on-board, and manage their product and services in the AWS cloud. This central interface and experience provides the gateway to unlock additional value that GoDaddy is looking to achieve by moving to AWS – ‘Accelerate time to market’, ‘Raise the bar on engineering rigor and quality’, ‘Leverage leading edge capabilities including machine learning’, ‘global scale and performance’, and achieving ‘velocity and flexibility that drives more agility’.
GoDaddy drove the creation of a great developer experience through a set of goals for the Public Cloud Portal initiative:
- Automate the process in such a way that allows GoDaddy to iterate and improve – Lowering the TCO for the onboarding process over time
- Communicate all the steps involved, manage the workflow, and manage the stepwise movement through the on-boarding process for teams
- Provide a central place to request, track, and view budgets versus actual spend
- Provide a wizard interface and automation for the AWS onboarding process, building in standards, guardrails, and best practices
- Provide curated learning paths aligned to the infrastructure architecture pattern the team selects for deployment
- Onboard the team to S-P-A-Q metrics collection and reporting – the team connects their application with the standard metric collection end-points for Speed, Performance, Availability, and Quality
- GoDaddy engineering teams can easily test an innovative idea. They can request an experimental budget and account, and begin building the same day.
Meeting these goals is critical to the success of the GoDaddy cloud adoption journey, enabling engineering teams with a one-stop shop to support, coordinate, and manage their every step along the way, including on-going operations.
Genesis of the Public Cloud Portal
So, how is GoDaddy realizing the benefits and objectives of moving to the cloud, scaling across 1000’s of employees, 100’s of scrum teams, and creating an experience that accelerates engineering teams in serving their customers? Managing deployment standards, setting up the cloud foundation and landing zones, organizing and collecting on-boarding information, tracking and reporting; is all too much to handle manually while supporting the scale and agility that is required. Enter the GoDaddy Public Cloud Portal, with the mission to deliver a seamless one stop shop for GoDaddy developers to learn, on-board, and manage their products and services in the cloud. Let’s dive into the feature/functions brought together in the GoDaddy public cloud portal in support of development teams in the next section.
The Public Cloud Portal ecosystem
The Public Cloud Portal is a one-stop shop for development teams and provides a consistent experience for all aspects of the teams’ product lifecycle including a managed workflow for on-boarding. The diagram below depicts all of the feature/functions brought together in the public cloud portal in support of development teams.
The One-stop Shop for product development ‘DevSecFinOps’ teams:
Here’s an overview of the main feature/functions shown in the diagram that GoDaddy provides to teams:
Budget and Finance
The portal provides an integrated approach for teams to create and manage budget requests, receive billing alarms and view their spending via a widget. Supporting tools are the AWS Billing and Cost Management Console and AWS Pricing Calculator. Teams take financial responsibility for their development project(s), prepare and submit budget requests for experimentation, dev/test, or production environment projects on AWS. Increasing scrutiny and approval levels are required as the team approaches funding requests for their production environment. Once approved for the particular environment and development phase, the portal moves them to the next stage of the on-boarding workflow. During operations, the portal provides visibility with a view of the teams’ actual spend versus approved budget.
Accounts and Supporting Architecture Creation
An account creation wizard walks the team through all standard architecture deployment choices and collects operational information required to create the project teams’ baseline account structure and AWS service regional deployment/configuration. The team chooses a main architectural pattern which determines which portfolios will be included in the account setup. The portfolios in-turn determine which service products the team will have access to in their service catalog. AWS Service Catalog, AWS CloudFormation, and AWS Systems Manager are the main supporting services for the controlled and standardized deployment of GoDaddy landing zones for teams. In addition, on-going guardrail controls are implemented with AWS Config for the created AWS environment and accounts.
The team also selects the number Availability Zones to be setup, which regions to deploy to, and which standard VPC/CIDR configuration option they require. Finally, the team completes operational information related to team roles/security levels, communication channels (email distribution list, slack channel), and standard monitoring/logging requirements.
Once all input is complete on the portal, the background automation kicks in to provision what the team has specified. At completion the team receives an email letting them know the accounts have been created and ready to go. From there, the team can access the AWS interface via the GoDaddy AWS chicklet in OKTA, and assume standard AWS Identity and Access Management roles that are assigned to each account.
Cloud Readiness Review
The Cloud Readiness Review is a key governance function that validates compliance with the GoDaddy defined Must-Have’s and Should-Do’s list. The Must-Have’s and Should-Do’s list is similar to, and more accurately a super-set of, the AWS Well Architected framework and review requirements. The GoDaddy requirements for development teams include aspects covering Security, Application Architecture (Reliability and Performance Efficiency), Operational Readiness (Operational Excellence), Budget/Finance (Cost Optimization), and Compliance & Privacy. Before taking production traffic, teams must formally complete the Cloud Readiness Review to certify that their project meets the GoDaddy engineering standards and rigor set forth in the Must-Have’s and Should-Do’s document. The portal provides a central place for teams to answer qualification questions, reviewers to review team responses, along with tracking of overall team questionnaire completion, reviewer comments and approvals.
Process Documentation
The Portal provides a central, organized repository of all needed process documentation for development teams. From submitting the initial on-boarding request to supporting live production traffic, the portal provides a series of documented processes to assist teams. The AWS Enterprise Support Infrastructure Event Management (IEM) process is an optional process that is linked and referenced for teams as they move toward production go-live in their AWS environment.
Incident Management
The team has an integrated widget and feed on service availability with a process to manage and remediate issues that impact production operations and traffic. Integrated services include information from AWS Service Health Dashboard, Personal Health Dashboard, and Support Center.
Security and Compliance
The Portal provides teams with a comprehensive view of their security and compliance state within AWS. Security findings from the GoDaddy implementation of AWS Security Hub are surfaced in the portal so that teams have visibility of security and compliance findings they need to be handling to meet GoDaddy on-going security and compliance requirements.
KPI and Metrics
The portal automates on-boarding the team to GoDaddy standard SPAQ metrics and the standard incident management and team notification pipelines. It publishes a widget that represents the Speed, Performance, Availability, and Quality of the teams’ customer facing applications and their backend APIs. The portal also measures and provides metrics on the time it takes for teams to move through each of the onboarding steps. Measuring each of the steps is important to provide data to make decisions about where to implement process improvements.
Curated Learning
Finally, the Portal organizes a central repository of curated documents, videos, tutorials, and other learning links based on the various AWS Services in GoDaddy’s Service Catalog of products for teams. Teams get a focused view of curated learning that is targeted at the Service Catalog products linked to their chosen standard architectural pattern. AWS Documentation and AWS YouTube channel assets are referenced and organized for development teams by the Cloud Portal.
Steps to Production
Here’s a view of how the Public Cloud Portal manages the Steps to Production for Teams:
The Public Cloud Portal manages the workflow for teams as they progress through all requirements needed for on-boarding to the AWS public cloud. Many steps are managed directly by the portal experience, while others are merely tracked by the portal while the team completes an external process. The portal team continues to iterate, bringing more management and workflow functionality into the portal based on a prioritized backlog.
Steps 1-4 take the team through the series of project preparation, budgeting, privacy assessment, and readiness review required by the business in order to get the team building and utilizing cloud resources on AWS. The on-boarding checklist guides the team with all up-front preparation activities for on-boarding and meeting business requirements for moving to the cloud – such as Estimating costs for budget submission, Defining your On-call Group, defining Service accounts, along with email distribution list and Slack Webhook for team communications. The checklist promotes the team completing activities that can be done up-front so they can be ready to smoothly flow-through the on-boarding process and wizard that is coming up.
Budget submission and approval ensures that the project is approved by the business at a specific level of spend and financial commitment for the defined project operating horizon. Privacy Impact Assessment and Cloud Readiness Reviews ensure the project team’s proposed product or service is compliant with company compliance policies and standards, and safe to operate in the cloud environment.
The Privacy Impact Assessment is a tool which helps the team identify the most effective way to comply with their data protection obligations and meet individuals’ expectations of privacy. The goal is to identify and reduce privacy risks while allowing the aims of the project to be met whenever possible.
The Cloud Readiness Review is a process of measuring and reviewing engineering best practices, as defined in the GoDaddy Must Have’s and Should Do’s document, against pillars of public cloud constraints. It is used to ensure the bar around engineering rigor is being raised with each project. It also helps to create an environment of consistency of approach and teams meet or exceed company goals of Security, Performance, Availability, and Quality (SPAQ). The Cloud Readiness Review is a scaling set of questions and checks that scales as the team moves from Experimental to Development to Production.
Step 5 ‘Account Creation’ is where the Public Cloud Portal takes all cloud architecture, cloud product, and team information requirements/choices and invokes back-end automation to turn all of the requirements into an implemented multi-account landing zone with chosen architecture patterns, cloud services, security, audit, monitoring, logging, networking, etc… This step which could take several days to complete if done manually is completely automated and completes in under 2 hours allowing the team to quickly jump in and begin to build their application product or service.
Finally, Step 6 guides the team through the required pre-production process of conducting a vulnerability scan, an initial penetration test, and an optional AWS enterprise support Infrastructure Event Management (IEM) process. The team is then ready to move to full production, start taking production traffic, and move to operational support mode with on-going DevOps iteration in their product or service.
How it Works
When a GoDaddy DevOps team wants to get on-boarded to the AWS Public Cloud they may enter or move to one of 3 main stages of development; Experimental, Development, or Production. The first choice may be for initial Experimentation in the cloud environment where the team ultimately gets a ‘Dev-Private’ account with a high degree of flexibility in deployment of AWS services along with greatest Administrative privileges. After entering their project proposal ideas, the budget approval for this experimental account is easy for an initial standard funding level and 3-month duration, with only low-level management approval required to proceed. The concept is to make it easy for DevOps teams with a great idea to get an experimental budget and environment to innovate quickly on behalf of GoDaddy customers.
Here’s how it works. You’re an engineer and have a great idea? Simply enter your idea on the portal, request an experimentation budget, discuss and get approval from your approving manager, and you can run the account creation wizard on the portal and be up and building in the same day. It’s a great way to get started quickly with just an idea for a new product or service and start building with cloud native architectures. The portal not only enables the cloud engineering process, it’s driving cultural change, allowing any engineer in the company to go out and innovate on behalf of the customer very quickly. To deliver this advance in agility has required a huge step forward in collaboration and agreement from the business, IT, and Finance teams together.
Next, the team can move on from experimentation or may jump-in initially to service/product development, and request a Development budget and environment. This provides additional accounts such as ‘Dev’ and ‘Test’ to work within for their project. Development requires the team create a specific budget request based on their application architecture/requirements and development timeline on the Cloud Portal. They then must receive management approval (a higher level than with Experimentation budget) in order to move forward to this stage of development after acknowledging additional Must-Have’s and Should-do’s for the development phase. Basically, as the team progresses in their journey from Experimental, to Development, to Production, they essentially request additional budget to fund the operations of the new accounts needed for that stage of the development lifecycle.
Finally, the team can move on to a Production environment providing them with additional accounts such as ‘Stage’, ‘OTE’, and ‘Production’. A new production budget submission and approval is required on the Cloud Portal, along with completing additional requirements such as a more stringent Cloud Readiness Review process (Must-Have’s and Should-Do’s for Production) to assure the proposed workload is compliant with GoDaddy standards and ready to take production traffic.
From this point onward, the team continues to rely on the Public Cloud Portal for operational and financial management as their product or service continues to evolve. The GoDaddy Public Cloud Portal has enabled the overall experience for GoDaddy teams to ‘Build better products faster’, leveraging AWS products like AWS Service Catalog, AWS CloudFormation, and AWS Systems Manager. The portal delivers a seamless one stop shop for GoDaddy developers to learn, on-board, and manage their product and services in the AWS cloud. This central interface and experience manages the overall framework to unlock the value that GoDaddy is looking to achieve by moving to AWS. The key success factor was GoDaddy’s foresight to invest heavily upfront with a Cloud Center of Expertise (App Services Team) organization, and accompanying cloud platform standards and implementation to facilitate and accelerate GoDaddy Line of Business team cloud adoption.
Measuring Success
Looking at Application Performance, GoDaddy has automated the creation of over a thousand landing zones, expediting and expanding GoDaddy’s performance footprint around the globe. Improvements in Speed of Delivery has reduced the process to launch a Production workload by 58% in 2020. With Reliability and Availability there have been zero outages across 200 Production workloads in eight different regions spanning over two years of development. Measuring Contribution at Scale there have been over 1,750 code commits to GoDaddy Service Catalog Products by over 120 different software development engineers.
Bottom line, GoDaddy has achieved significant measureable success with the Public Cloud Portal and associated processes to date and is looking ahead for additional innovations and iteration as they accelerate their cloud adoption journey.