Building Your Own AWS Service Broker Services

The AWS Service Broker open source project, launched at re:Invent 2017, allows developers using application platforms such as Pivotal Cloud Foundry to provision and expose native AWS services from within application platform interfaces.

Though the Service Broker caters for a growing list of services, customers are looking for a simple way to extend AWS Service Broker services themselves. In this post we will explore what is happening in the background of the AWS Service Broker, and how development, operations, and security teams can customize solutions for their business requirements.

The process for creating customized services has changed since the first launch of the Service Broker, making it a simpler and more pleasant experience. We will cover some details on this, starting with how AWS CloudFormation is used by the Broker and how CloudFormation templates can be customized. We’ll then look at how parameter mappings between CloudFormation and the Broker have been simplified, and how a custom template catalog can be used to store customizations.

Service Broker and AWS CloudFormation

The AWS Service Broker uses AWS CloudFormation infrastructure-as-code templates to provision resources used within application platforms. For each service available in the Service Broker, there is a prescriptive template. AWS Service Broker templates are open source, so that you can modify and extend them to meet your needs.

In the initial release of Service Broker, you had to create a CloudFormation template for the service you wanted to use. We simplified the user experience by using default values for some of the inputs needed in the CloudFormation template at the time of provisioning. (Though these default parameter values suit the most common use cases, they can be overwritten as needed.)

Service Broker then needed a way to understand which parameters in the CloudFormation template should be presented within the application platform UI, which parameters are optional, which are required, etc. In the first iterations of the Broker, this required creating a YAML to document the mapping between the parameters in the CloudFormation template and the Broker. Now, you can instead store your own templates in a custom catalog and use automatically-generated Service Broker metadata instead of manual parameter mappings.

Store your own templates in a custom catalog

Service Broker itself now allows you to store templates in a custom catalog – you can completely customize the templates you use with the Broker. There are a few use cases where this is desirable.

Some customers are using their own existing CloudFormation templates within the Broker; often these are already in use as part of their existing infrastructure-as-code management, meet their requirements, and have passed their security assessments. For these customers, it makes sense to consume these existing templates within the Broker.

Some customers, especially those who have a strict security assessment process, want to use complex aggregations of templates (such as a VPC, database layer, and then a front end managed by a nested stack of templates) and provide these as reusable, self-service building blocks to development teams. DevSecOps teams can assess the existing templates, modify them to meet their business needs, and store white-listed, security-approved versions within their own catalog. This will require an S3 bucket to store the templates, and the Broker’s config YAML file will need to be updated to list the bucket location. This is also required for customers using the AWS Service Broker with AWS GovCloud, as systems running in GovCloud require their resources to be stored in Amazon S3 within the GovCloud region, and cannot consume the existing Service Broker templates from the public S3 catalog.

For more details on setting up the catalog, see the AWS Service Broker docs. There is even a CloudFormation prerequisite template which you can use to build out the S3 bucket and required permissions, launch the stack, and add the info within the outputs to the broker config.

Example config for OpenShift

(Note the S3 bucket and Region configuration.)

TARGETACCOUNTID=
TARGETROLENAME=
VPCID=
REGION=us-east-1
IMAGE=awsservicebroker/aws-servicebroker:beta
IMAGEPULLPOLICY=Always
S3BUCKET=<YOUR S3 BUCKET IN GOVCLOUD>
S3KEY=templates/latest
S3REGION=us-gov-west-1
TABLENAME=awssb
VERBOSITY=10
BROKERID=awsservicebroker
PRESCRIBE_OVERRIDES=true

Example config for Cloud Foundry

Moving from manual parameter mappings to metadata

The second change we made was to reduce the difficulty of the parameter mapping process. Instead of requiring a separate YAML file for the mappings, the Broker has been updated to generate this mapping from metadata within the CloudFormation template.

...
Metadata:
  AWS::ServiceBroker::Specification:
    Version: 1.0
    Tags:
    - AWS
    - Lex
    - Bot
    - NLU
    - chatbot
    Name: lex
    DisplayName: Amazon Lex
    LongDescription: Amazon Lex is a service for building conversational interfaces
      into any application using voice and text. Amazon Lex provides the advanced
      deep learning functionalities of automatic speech recognition (ASR) for converting
      speech to text, and natural language understanding (NLU) to recognize the intent
      of the text, to enable you to build applications with highly engaging user experiences
      and lifelike conversational interactions.
    ImageUrl: https://s3.amazonaws.com/awsservicebroker/icons/AmazonLex_LARGE.png
    DocumentationUrl: https://thinkwithwp.com/documentation/lex/
    ProviderDisplayName: Amazon Web Services
    ServicePlans:
      default:
        DisplayName: Standard
        Description: Amazon Lex
        LongDescription: Creates an Amazon Lex bot
        Cost: https://thinkwithwp.com/lex/pricing/
        ParameterValues: {}
...

You can see that the Broker specification defines a description, service plans, required or optional parameters, and defaults. This specification can be very simple, as in the example here, or more complex, as in the case of Amazon Relational Database Service (Amazon RDS), where there are different plans for dev and prod with rather different parameters and options.

Moving the mapping info into a meta section of the CloudFormation template allows for a single source of truth, making it simpler for customers to maintain customizations. But the creation of the metadata was still a burden.

Feedback from a customer engineer was that metadata creation was painful, prone to error, and you really needed to keep your wits about you when working with multiple plans. Depending upon the complexity of the parameter options in your CloudFormation template, you could easily get lost, forget, or overlook something. This engineer’s most telling comment was that the time and effort needed to build the metadata by hand was keeping him away from building the new Lego lunar module!

Automatically generating the Service Broker metadata

Simple scripting to the rescue! We now offer a metadata generator to help build customizations. This allows you to build out or customize the CloudFormation template, and run the metadata generator which builds out the metadata based on the parameters in the template. The output from the metadata generator can then be placed in the CloudFormation template, and the new template added to your custom catalog S3 bucket.

Example output from the metadata generator

...
==================================YAML========================================

AWS::ServiceBroker::Specification:
  Version: 1
  Tags:
  - ''
  - ''
  Name: ''
  DisplayName: ''
  LongDescription: ''
  ImageUrl: ''
  DocumentationUrl: ''
  ProviderDisplayName:  Amazon Web Services
  ServicePlans:
    production:
      DisplayName: Production
      Description: Configuration designed for production deployments
      LongDescription: ''
      Cost: ''
      ParameterValues: &o0
        AccessCidr:
        AllocatedStorageAndIops: 100GB 1000IOPS
        AllowMajorVersionUpgrade: false
        AutoMinorVersionUpgrade: true
        AvailabilityZones: Auto
        BackupRetentionPeriod: 35
        CidrBlocks: Auto
 ...

Depending on the application platform where the Service Broker is used, you may need to perform an action for the Broker to see the new template. Application platforms such as Red Hat OpenShift will poll for changes and update after some time. Pivotal Cloud Foundry only loads the list of templates at install; if you are using Pivotal Cloud Foundry, you will need to do a deploy of the Broker. This will not affect any existing services provisioned by the Broker, as this info is stored in a DynamoDB stable session state.

To reiterate, you need to:

• Create an S3 bucket to store the customized or new service templates.
• Update the Service Broker config to use the catalog bucket (see example configs for OpenShift and Cloud Foundry, above).
• Create or customize the CloudFormation template, and ensure that you have all the parameters and outputs needed to launch and consume the service for your business needs.
• Use the metadata generator to build out the Service Broker specification.
• Copy the output from the generator and insert it into the CloudFormation template.
• Upload the template with its metadata spec to the catalog S3 bucket used for testing.
• Create the new service per the application platform you are using, e.g.: cf create-service <newcustomservice> default <blognewcustomservice>.

Is this process absolutely necessary? No, plenty of customers are consuming the existing prescriptive  Service Broker services. Customers on AWS GovCloud are already using the custom catalog, but most are taking the existing prescriptive templates, downloading them, then uploading them to their own custom catalogs. Customers looking to build their own custom or aggregate solutions to be delivered by the Service Broker will find this process simpler.

Next steps

• Go try it out yourself!
• The AWS Service Broker is an open source project – feel free to contribute in extending the AWS Service Broker, service templates, or simply raise issues under the Service Broker project.

More details are available about the AWS Service Broker.