Announcing Cloud Custodian Integration with AWS Security Hub

中文版

One of the popular options for automated security, compliance, and cost management solutions in the cloud is Cloud Custodian, an open source project sponsored by Capital One. Cloud Custodian is a flexible rules engine with policies written in simple YAML configuration files, with support for over 144 AWS resource types. However, because Cloud Custodian reports primarily via email notifications or command line output, and cloud watch metrics, the lack of a consolidated view with search and filtering capabilities is a pain point for customers who have large environments and multiple accounts.

That pain is now addressed by the integration of Cloud Custodian as a Findings Provider to AWS Security Hub. The newly-announced AWS Security Hub provides a comprehensive view of your security state within AWS, and your compliance with security industry standards and best practices. Security Hub collects security data from across AWS accounts, services, and supported third-party partners to help analyze security trends and identify the highest-priority security issues.

Solution Overview

This solution will cover how to upgrade and configure an existing Cloud Custodian environment to post (aka, send) findings into AWS Security Hub, then view those findings in Security Hub. If you don’t already have a working Cloud Custodian environment, please visit Cloud Custodian’s Getting Started page.
Security Hub must already be activated in the account.
The example will show Cloud Custodian detecting S3 buckets that don’t have any bucket policy statements that enforce encryption. If you don’t have any such buckets, just create an empty S3 bucket without any encryption policy, or use your own S3 or EC2 policy which you know will trigger a finding.
The AWS credentials used to execute Cloud Custodian need to allow the action securityhub:BatchImportFindings to permit the import.

Step 1: Upgrade Cloud Custodian

From within an existing Cloud Custodian virtual environment, run the following command:

pip install c7n --upgrade

Step 2: Set Required AWS IAM Permissions

The action securityhub:BatchImportFindings is required. If this action is not already allowed by the credentials which will be used, either add them to an existing customer-managed policy, or attach the new AWSSecurityHubFullAccess AWS Managed policy. Note: IAM Users or Roles with either arn:aws:iam::aws:policy/AdministratorAccess or arn:aws:iam::aws:policy/PowerUserAccess already have the required permissions.

Step 3: Create a Cloud Custodian Policy with the new action

The enhancement to Cloud Custodian is the addition of the action type “post-finding.” The following is an example of the minimum policy changes required to get a finding to import:

    - type: post-finding
        severity_normalized: 30
        types:
          - "Software and Configuration Checks/AWS Security Best Practices"

Cut and paste the following example of a minimal policy to a new file named s3-no-encryption.yml

policies:
  - name: s3-bucket-encryption-policy-absent
    resource: s3
    description: s3 buckets without encryption required 
    filters:
      - type: no-encryption-statement
    actions:
      - type: post-finding
        severity_normalized: 30
        types:
          - "Software and Configuration Checks/AWS Security Best Practices"

For any policies deployed as a Lambda, which means any policies with a “mode” property, the following line must currently be added within the mode section of the policy as the current default lambda runtime is currently using a version of the boto libraries which do not support securityhub

       packages: [boto3, botocore, urllib3]

Step 4: Run the Policy

The following command will execute Cloud Custodian using the policy file s3-no-encryption.yml in the local directory. Cloud Custodian supports the the same settings as the AWS CLI for security credentials and regions.

custodian run -s /tmp s3-no-encryption.yml

The result of that run command should be similar to the following, with expected differences being timestamps, execution_time, and the count and number of resources. If the count in the first line is zero, then the second line won’t be present as there were no findings to be sent to Security Hub.

2018-11-19 05:43:30,095: custodian.policy:INFO policy: s3-bucket-encryption-policy-absent resource:s3 region:us-east-1 count:22 time:5.80 
2018-11-19 05:43:30,832: custodian.policy:INFO policy: s3-bucket-encryption-policy-absent action: bucketfinding resources: 22 execution_time: 0.73

Step 5: View the Finding in the Security Hub Console

Log in to the AWS Console, select Security Hub, then, in the Navigation area in the top left, click Findings.
In the Findings area, click in the Filters input window, then, in the selection list for Filters, select Product Fields. For Key enter “ProviderName”, and Value enter “CloudCustodian”.
Observe how the Name in the Cloud Custodian policy mapped to the Title of the finding. The Description in the policy maps to the Description in the finding, which is displayed in the detail panel below the Finding ID.
To see the complete set of data that Cloud Custodian included as part of the finding, click the link to the right of “Finding ID:”

Step 6: Exploring Additional Data Which Can Be Passed

To view the additional properties and allowed values for the “types” field that Custodian supports submitting to Security Hub, run the following command:

custodian schema ec2.actions.post-finding

The property names map directly to fields defined in the standard findings format used by Security Hub.

Step 7: Exploring Which AWS Resource Types are Supported

Update: The 0.8.33 release on 1/1/2019 provides the post-findings action for all supported AWS resource types, thus the following section in no longer relevant.

At the launch of this integration, the following AWS resource types (as defined in CloudCustodian) are supported: ec2 and s3. All other AWS resources types supported by CloudCustodian have post-finding action support in the version of CloudCustodian master branch on github that can be installed using a Cloud Custodian Developer Install procedure. However this additional resource support is not yet released to pypi.org, thus the “pip install –upgrade c7n” from step #1 of this post will still only support the ec2 and s3 resources for post-findings. To see if a specific resource has had post-finding action support added, run:

custodian schema <resource>.actions.post-finding

~~If you get the following error message, it means Cloud Custodian does not support integration with that resource as yet:~~

custodian.commands:ERROR post-findings is not in the actions list for resource

Step 8: Exploring the Code and How to Contribute

The source code for Cloud Custodian’s integration is contained in securityhub.py where the function process() calls the Security Hub public API named BatchImportFindings. In the same file, the function OtherResourcePostFinding contain the resource level data mapping for non ec2 and s3 resource types. Please submit at PR or submit an issue with a title prefixed with ‘[SecurityHub]’ if you find any issues. Also, please submit a PR or issue for any desired additional data mappings to the Amazon Finding Format. You can also join the Cloud Custodian community in their chat room on gitter.im, with hundreds of other users and developers that can help get you started.

The primary reason for this code contribution to Cloud Custodian is to support AWS customers already using Cloud Custodian. Other open source security and compliance assessment projects can also use this as an example of how to integrate with AWS Security Hub.

AWS Open Source Blog