Integrate AWS Cost Anomaly Detection Notifications with IT Service Management Workflow – Part1 Jira

AWS Cost Anomaly Detection uses advanced Machine Learning to identify anomalous spends and generate timely alerts, empowering our customers to take quick action. To assign, track, and manage the efforts, teams following the IT Service Management (ITSM) framework may need duly assigned incident tickets.

In the era of agile practices, ITSM processes have evolved into advanced yet flexible and customizable workflows. By leveraging automation, we can invoke high-velocity workflows as mechanisms to create tickets, add context, and enable the responders to resolve issues rapidly. This also benefits by eliminating any potential scope for human errors and reducing administrative overhead. You remain in sync with your organization’s defined mechanisms to document and track the end-to-end flow.

This post explains how we can integrate AWS Cost Anomaly notifications with Jira Service Management, leveraging Jira cloud automation to document, review, and resolve the AWS cost anomalies in adherence with the defined ITSM framework.

Architecture Overview

The following architecture diagram illustrates the use of AWS Lambda to retrieve the cost anomaly alerts and in turn, call a service endpoint using incoming webhooks in Jira Service Management. When a cost anomaly is detected by AWS, the cost monitor generates an alert notification and publishes it to an Amazon Simple Notification Service (SNS) topic. The SNS topic has a subscription for Lambda function which retrieves the message and calls an incoming webhook URL endpoint in Jira automation, also supplying the notification message. Optionally, you can create an SNS topic subscription to send email/SMS notifications to cloud admins. The Jira automation rule then triggers the workflow to create a new issue in the Jira SM project, along with the appropriate field mappings that you’ve defined.

Figure 1. architecture overview

Prerequisites

1. AWS Cost Anomaly Detection is a feature within AWS Billing and Cost Management Console. To access AWS Cost Anomaly Detection, enable AWS Cost Explorer first. For instructions on how to enable AWS Cost Explorer using the console, see Enabling AWS Cost Explorer.
2. Configure AWS Cost Anomaly Detection so that it detects anomalies at a lower granularity and spend patterns, in context to your monitor type. Creating your cost monitors
3. Create an AWS Simple Notification Service (SNS) Topic with the required permissions for the cost anomaly to publish the alert notifications to. Creating an Amazon SNS topic for anomaly notifications
4. Configure Jira Service Management project within which the automation workflow creates the new issue. Jira quick start guide

Implementation Steps

Step 1: Generate an incoming webhook URL and configure automation workflow in Jira Service Management

1. Login to the Jira service management console with administrative privileges and go to System Settings
2. Choose Global Automation from the left pane and choose Create rule
3. Under Add a trigger > Integrations, choose Incoming webhook
4. This generates a new Webhook URL. Copy the URL for use in next step  Refer: Automation webhooks
   

   Figure 2. Incoming webhook

5. Under “Execute this automation rule with”, choose “No issues from the webhook” and Save
6. On the next step for “Add component”, choose “Add an action” > “Create issue”
7. Select your project and the Issue type
8. Under Summary, provide a summary text for your issues including optional Jira rule smart valuesExample: “AWS Cost Management: Anomaly Detection AWS Account {{webhookData.accountId}}”
9. Under Description, provide this configuration text code to map the cost anomaly notification JSON data with Jira workflow using smart values
   Note: Modify the text and smart values as per your use-case
10. Choose Next and turn this rule on

Step 2: Launch the AWS CloudFormation Template

Download and launch this AWS CloudFormation template to deploy an AWS Lambda function, Amazon CloudWatch Logs group, and add Lambda subscription to the SNS topic.

Note: Anomaly notification published to the SNS topic, followed by the Lambda function trigger can incur costs. To learn more about pricing refer: AWS Lambda pricing; Amazon SNS pricing; AWS Pricing Calculator.

1. Sign in to the AWS Management Console and navigate to the AWS CloudFormation console
2. Choose Create Stack > “With new resources” and specify a “Stack name”
3. Under “Parameters” provide the JiraWebHookURL you had created in Step 1, the SnsTopicArn for your SNS topic (created as part of pre-reqs) and choose Next
   

   Figure 3. Specify stack details

4. Leave the “Configure stack options” at default values, add tags as needed and choose Next
5. Review the details on the final screen and under “Capabilities” check the box for “I acknowledge that AWS CloudFormation might create IAM resources with custom names”
6. Choose Submit
   Note: You can review the progress of your new stack under AWS CloudFormation > Stacks > [StackName] > Events tab
   Once the Stack is created successfully, you will see the following resources deployed:
   AWS Lambda function, Amazon CloudWatch Log group, SNS Subscription for the Lambda function under your SNS Topic and the corresponding AWS IAM Roles and Policies.

Step 3: Test your Jira automation rule using a sample JSON

1. Sign in to the Amazon SNS console
2. In the left navigation pane, choose Topics
3. Choose the SNS topic you had created as the pre-requisite, and then choose Publish message
4. In the Message body section, copy this test JSON and choose Publish message
5. On successfully publishing the message to the SNS topic, the Lambda function triggers the incoming webhook and in turn triggers the automation workflow in Jira SM
   Note: You can verify the Lambda execution using CloudWatch metrics under the “Monitor” tab
6. The Jira automation rule creates the new issue/task under the Jira project you had specified
   

   Figure 4. Jira creates new issue_task

7. When triggered by the cost anomaly alert notification, the rule creates the new task furnishing all the details from the cost anomaly notification fields
   

   Figure 5. Creates new tasks with details

Clean up

If you would like to clean up the resources deployed for this post, please refer the following:

1. Delete AWS CloudFormation Stack
2. Delete Amazon SNS topic
3. Delete or disable the Jira automation rule
4. Delete the incoming webhook URL

Conclusion

In this blog post, we discussed how we can integrate the AWS Cost Anomaly notifications with ITSM tools like Jira service management to trigger automation workflows using incoming webhook URLs. This empowers you with an efficient mechanism to automate the end-to-end flow of work to identify, document and track issues related to anomalous spends in your AWS account. To get started and learn more, visit Getting started with AWS Cost Anomaly Detection and Jira quick start guide.

As an option, you can also add subscriptions for other protocols like Email, SMS, HTTPS etc. to the SNS Topic to get notifications and/or trigger additional workflows. For example, you can integrate the solution to visualize the cost anomaly data using Amazon Managed Grafana. Alternatively, in order to have an organizational view for the cost anomaly data in Amazon QuickSight, you can deploy the Cost Anomaly Dashboard.