Understand and Optimize AWS Data Transfer Charges for Splunk Cloud on AWS Ingestion

By Ranjit Kalidasan, Partner Solutions Architect – AWS
By Karsten Ploesser, Principal Solutions Architect – AWS

Splunk

Amazon Web Services (AWS) customers using Splunk Cloud on AWS for their security, operational, and observability use cases may manage large volumes of data. Having a thorough understanding of AWS data transfer charges can help them optimize their architectures and costs.

This post discusses the data transfer costs for five of the most common Splunk use cases. Other AWS service charges are out of scope for this post but should also be considered when designing any architecture.

The content of this post is only applicable to the Splunk Cloud Platform service available in AWS regions. For a full list of AWS regions in which Splunk Cloud is available, please consult Splunk’s documentation.

Splunk is an AWS Partner with multiple Competency designations including Data and Analytics and Cloud Management Tools. Splunk is one of the world’s first data-to-everything platform, and removes barriers between data and action.

Common Splunk Data Ingest Scenarios

At AWS, we talk to hundreds of customers every year about selecting their strategy to get data into Splunk Cloud on AWS. These are the five most common scenarios many of our customers employ for scalable data ingestion within and across AWS regions.

Note that most customers deploy more than one of these for their Splunk use cases.

Scenario #1: Using Splunk Forwarders from Public Subnet and Internet Gateway

In this scenario, Splunk Forwarders in a public subnet within an Amazon Virtual Private Cloud (Amazon VPC) use an Internet Gateway (IGW) to send data to the Splunk Cloud endpoint. The Amazon VPCs are in the same AWS region as the Splunk Cloud endpoint (for example US-East-1 to US-East-1).

Figure 1 – Splunk Forwarders from public subnet – same region.

Scenario #2: Using Splunk Forwarders from Private Subnet and NAT Gateway

In this scenario, Splunk Forwarders in a private subnet belonging to an Amazon VPC use a network address translation (NAT) gateway to send data to the Splunk Cloud endpoint. The Amazon VPCs are in the same AWS region as the Splunk Cloud endpoint (for example US-East-1 to US-East-1).

Figure 2 – Splunk Forwarders from private subnet – same region.

Scenario #3: Using Splunk Forwarders from Public Subnet and IGW Across Regions

This scenario is similar to the first one shared above, except that the Amazon VPC containing the forwarders is in a different AWS region than the Splunk Cloud endpoint.

For example, the forwarders in US-East-1 (Virginia) forward data to Splunk Cloud in US-West-2 (Oregon). Learn more about the AWS regions where Splunk Cloud is available.

Figure 3 – Splunk forwarders on public subnet – cross-region.

Scenario #4: Using Splunk Forwarders from Private Subnet and NAT Gateway Across Regions

This scenario is similar to the second one shared above except that traffic crosses AWS regions.

Figure 4 – Splunk forwarders on private subnet – cross-region.

Scenario #5: Ingesting Data Using AWS Services

In this scenario, data is sent via Amazon Kinesis Data Firehose into Splunk. Kinesis Data Firehose supports Splunk as one of the partner destinations.

In the example shown above, Kinesis Data Firehose sends AWS CloudTrail data with Amazon CloudWatch logs as a destination, and uses Kinesis Data Firehose to send this data to Splunk with a subscription filter configured in CloudWatch Logs.

This architecture uses the Splunk HTTP Event Collector (HEC). Although the figure depicts the use of cross-region endpoints, the data transfer charges are the same for both cross-region and same-region Splunk Cloud endpoints.

Figure 5 – Splunk data ingestion using Amazon Kinesis Data Firehose.

What’s Driving Data Egress Charges in Source Account(s)?

Keeping costs low is a top priority for most customers. Being aware of data egress costs helps you get the most out of Splunk, weigh the costs of each ingestion scenario, and pick the one that’s right for you.

Data transfer costs are covered comprehensively in this AWS blog post. Here are the main items that make up data transfer costs in Splunk data ingest scenarios:

• Data transfer – same region: You are charged data transfer charges when sending traffic from an Amazon Elastic Compute Cloud (Amazon EC2) instance in one VPC to a public IP associated with another VPC in the same AWS region. This charge applies to Splunk data ingest scenarios 1 and 2.
• Data transfer – cross-region: As in the case of same-region data transfer, you are charged for data traffic crossing AWS regions. This charge applies to Splunk data ingest scenarios 3 and 4.
• NAT Gateway charges: NAT Gateway charges fees for each hour a gateway is active and incremental charges for the amount of volume processed by the gateway. This applies to Splunk data ingest scenarios 2 and 4.

Let’s look at the applicability of data transfer charges for each scenario. Costs differ between same-region (use cases 1 and 2) and cross-region data transfer (3 and 4) and whether data transfer flow via Internet Gateway (1 and 3) or NAT Gateway (2 and 4).

Amazon Kinesis Data Firehose (use case 5) does not distinguish between same-region or cross-region transfer and uses a data processing charge (price per GB data processed) instead.

Optimizing Data Transfer Cost in the Context of Splunk

Data transfer cost optimization involves adjusting your data ingestion strategy to minimize the charges you incur.

Based on best practices and our experience gained from working with Splunk customers, we recommend considering the options below to minimize data transfer costs:

• Select a Splunk Cloud endpoint in the same AWS region as your workload. Keep traffic local to an AWS region in order to optimize your data transfer costs. If you’re in an AWS region that does not provide an endpoint, or you have other requirements such as a single, centralized Splunk endpoint, cross-region data transfer costs could come into play.
• Depending on your security rules and configuration, configure your instances to have public IP addresses, which allows you to bypass or remove the NAT Gateway. You’ll need to place Splunk Forwarders into a public subnet with access to an IGW that’s part of your Amazon VPC in order for this to work.

Security is job zero. When considering changes such as assigning public IPs to Splunk Forwarders, take security policies into account. For example, configure Amazon VPC security constructs including security groups and Network Access Control Lists (ACLs) to reduce the attack surface of your Forwarder deployment. This enables you to optimize data transfer cost (by removing NAT Gateway) without compromising your security posture.

Sample Data Transfer Calculations

The sample calculations in this section are provided for the purpose of illustration and are subject to change as AWS pricing is updated. Please refer to the Amazon EC2 pricing page or the AWS Pricing API for up-to-date pricing information. As baseline to our sample calculation, and in order to explain the charges for each use case, let’s take the example of a 1 TB data to be ingested into Splunk Cloud.

Use Case #1: Using Splunk Forwarders from Public Subnet and IGW

In this example, we assume sending 1 TB of data from the AWS region US-East-1 (North Virginia) to a Splunk Cloud endpoint in same region.

▶ Data transfer charges: 1,000 * $0.01/GB = $10 USD

Use Case #2: Using Splunk Forwarders from Private Subnet and NAT Gateway

In this example, we assume sending 1 TB of data from a private subnet with NAT Gateway from the AWS region in North Virginia to a Splunk Cloud endpoint in same region.

▶ Data transfer charges: 1,000 * $0.01/GB + 1,000 * 0.045 = $10 + $45 = $55 USD

….* There will be also additional NAT Gateway usage charges by hour applicable.

Use Case #3: Using Splunk Forwarders from Public Subnet and IGW Across AWS Regions

In this example, we assume sending 1 TB of data from the AWS region US-East-2 (North Virginia) region to a Splunk Cloud endpoint in the AWS region US-West-2 (Oregon).

▶ Data transfer charges: 1,000 * $0.02/GB = $20 USD

Use Case #4: Using Splunk Forwarders from Private Subnet and NAT Gateway Across AWS Regions

In this example, we assume sending 1 TB of data from a private subnet with NAT Gateway from the AWS region US-East-1 (North Virginia) to a Splunk Cloud endpoint in the AWS region US-West-2 (Oregon).

▶ Data transfer charges: 1,000 * $0.02/GB + 1,000 * 0.045 = $20 + $45 = $65 USD

….* There will be also additional NAT Gateway usage charges by hour applicable.

Use Case #5: Ingesting Data Using AWS Services

In this example, we assume sending 1 TB of data from Amazon Kinesis Data Firehose to a Splunk Cloud endpoint hosted in the AWS region US-West-2 (Oregon). Kinesis Data Firehose pricing is based on 5KB rounding on the incoming records. To get an approximate ingestion of 1TB, we assume an 80 records per second ingestion with 5KB size per record.

Unit conversions management events:

• Number of records for data ingestion: 80 per second * (60 seconds in a minute x 60 minutes in an hour x 730 hours in a month) = 210,240,000 per month

Pricing calculations:

• 210,240,000 monthly records x 1 unit multiplier = 210,240,000 total records per month
• RoundUp (5) = 5 KB average record size (rounded to nearest 5 KB increment)
• 210,240,000 records x 5 KB = 1,051,200,000.00 KB per month
• 1,051,200,000.00 KB / 1,048,576 KB in a GB = 1,002.50244140625 GB per month
• Tiered price for: 1,002.50244140625 GB
• 1,002.50244140625 GB x 0.0290000000 USD = 29.07 USD
• Total tier cost = $29.07 USD for data ingestion
• Additional data transfer charges: $0 USD

Conclusion

Data transfer cost is a key component to consider when selecting your strategy to get data into Splunk Cloud on AWS.

We talk to hundreds of customers implementing Splunk Cloud on AWS each year about optimizing their deployments. Cost optimization and selecting the right data ingestion strategy rank among their top priorities. Use this post to understand what drives data transfer cost and choose the best strategy for your situation.

We understand that no single workload is the same and you may have specific requirements that prevent you from implementing a particular strategy or cost optimization recommendation.

Contact your Splunk and AWS account teams to get help in this case. This post represents the best practices at the time of writing and will be updated as needed so that AWS and Splunk customers continue to get the most out of both platforms in a cost-effective way.

.

.

Splunk – AWS Partner Spotlight

Splunk is an AWS Partner with multiple Competency designations including Data and Analytics and Cloud Management Tools. Splunk is one of the world’s first data-to-everything platform, and removes barriers between data and action.

Contact Splunk | Partner Overview | AWS Marketplace