Security is time series: How VMware Carbon Black improves and scales security observability with Amazon Timestream

August 30, 2023: Amazon Kinesis Data Analytics has been renamed to Amazon Managed Service for Apache Flink. Read the announcement in the AWS News Blog and learn more.

Amazon Timestream is a fast, serverless, and secure time series database and analytics service that can scale to process trillions of time series events per day. Organizations are dealing with an increasing amount of security data, generated in logs and events, needed to quickly and effectively address potential security threats. Because security data (logs, metrics, events, and so on) is inherently time-based, a time series solution like Timestream provides the right tools for collecting, storing, and analyzing this data for threat detection, behavioral analytics, and other security use cases.

For many of our customers, security threats are constant and expensive to remediate, so it’s no surprise that security monitoring is one of their highest priorities. In 2022, data breaches cost $4.35 million and took 277 days to identify and contain on average, according to IBM Research. The challenge in detecting security threats is often finding anomalies in large volumes of data. Customers face increasing operational overhead, scalability constraints, and high costs when dealing with large amounts of data. With the right tools, you can effectively use data to secure your applications. Timestream helps strengthen your security posture through its ability to manage, analyze, and visualize time series data at scale.

Timestream supports a number of security use cases, including threat detection, behavioral analytics, and incident investigations, because data used in these use cases is inherently time-based. With its integrations and low-effort onboarding, Timestream makes it simple to start ingesting and analyzing large volumes of time series data in real time so you can build the core components of your security analytics workflow: data ingestion, storage, analytics, visualization, and alerting.

VMware Carbon Black is a leader in cloud-native endpoint security. In this post, we describe how Carbon Black uses Timestream for observability in their security products, and how other security customers use Timestream to build core components of their security analytics workflows.

Improving security observability with Timestream

Security teams sometimes lack the right tools to gain insights and improve the performance of their applications and infrastructure. Poor query performance, high infrastructure management overhead, and rising costs are only a few of the challenges that keep security teams from effectively securing their organizations.

With Timestream, Carbon Black successfully navigated these challenges by implementing an observability solution for their security application. Their application generates customer bandwidth metrics that are used for monitoring the thousands of customers whose data is ingested via Amazon Kinesis Data Streams. Timestream ingests these customer bandwidth metrics and powers the analytics Carbon Black uses to automatically scale their Kinesis streams in real-time. Their Kinesis scaling solution ensures there is enough capacity to handle the amount of traffic they receive for each of their customers, while also reducing costs by appropriately scaling down.

“Implementing Timestream allowed us to track some high cardinality metrics we needed to perform automated custom scaling with Amazon Kinesis Data Streams. Other products challenged us with query times, cost, or system reliability, but for our use case, Timestream offered improvements for all three. After implementing Timestream, we no longer need to think about this component in our system. Like any good embedded product, it just works.”

— Corey Leopold, Staff Engineer at VMware Carbon Black

Timestream also just works at scale, and has allowed Carbon Black to ingest and query customer bandwidth metrics simultaneously for large workloads. The ingestion layer in Timestream automatically scales, which allows them to ingest high cardinality data from thousands of customer containers that each contain thousands of endpoints. In addition, because the Timestream ingestion layer is decoupled from its query layer, Carbon Black analyzes and acts on customer bandwidth metrics with virtually infinite scale. They also use the Timestream Query Editor to run performant, one-off queries, and to model their queries before deploying into production.

Their query results are plugged into their existing data pipelines and visualization applications using Timestream native integrations like AWS Lambda and the JDBC connector. Moreover, because Timestream is serverless, there is no need for them to manage additional infrastructure or provision capacity. This enables Carbon Black to continuously monitor their own applications even through usage spikes without additional overhead.

Now that we’ve explored how Carbon Black uses Timestream, the rest of this post will provide a high-level overview of how other security teams use Timestream to build their security analytics solutions.

Solution overview

The following diagram illustrates our solution architecture for other security applications.

The following sections describe how to achieve full visibility of your security data so that you can take action on security threats, address anomalies, and identify malicious trends.

Ingest data into Timestream

To generate full visibility of your security data, you need to gather data that isn’t always located in the same systems or applications. Having the ability to ingest data from wherever it resides is crucial in building your security toolset. With the flexible schema in Timestream, it’s simple to aggregate data across many security activities that occur over time. Additionally, the ingestion layer in Timestream automatically scales, which allows you to ingest data from multiple sources. This enables you to gather intelligence across different data sources and build comprehensive profiles of security threats, compromised accounts, or other malicious patterns.

There are a number of ways to bring your data into Timestream, such as through its APIs (single data points or batched data), or through its several integrations like Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK), and Telegraf. For more information, refer to Amazon Kinesis, Amazon MSK, and open source Telegraf, respectively. For migrating larger amounts of data or lift-and-shift workloads, use batch load in Timestream to ingest files from Amazon Simple Storage Service (Amazon S3) directly into Timestream without having to rely on other tools or write custom code.

For a sample application for ingesting data from Kinesis to Timestream, refer to Amazon Kinesis Data Analytics for Apache Flink.

Store and manage data with data lifecycle policies

Your data is automatically encrypted using either AWS Key Management Service (AWS KMS) or your own keys, so you don’t need to manually encrypt data at rest or in transit. For more information, refer to Key management.

Timestream helps ensures durability of your data by automatically replicating your data across different Availability Zones within a single AWS Region. Timestream enables you to configure retention policies to automatically assess your storage tier, and move data to a more cost-effective option as needed, without losing the query performance you expect. Additionally, you can set a storage retention value anywhere from 1 hour to 200 years to permanently delete data after this storage duration. This means you can use the memory tier for fast point lookups, while also using the magnetic tier for analytical queries as well as data aging and deletion. We explore these different query patterns in the next section.

Lastly, Timestream offers further protection of your data through a native integration with AWS Backup. Data backup in Timestream is a fully managed feature that simplifies the creation, migration, restoration, and deletion of backups, while providing improved reporting and auditing. Through integration with AWS Backup, you can use a fully managed, policy-driven centralized data protection solution to create immutable backups and centrally manage data protection of your application data spanning Timestream and other AWS services supported by AWS Backup. For more information, refer to Working with AWS Backup.

Visualize your security posture

With data now available in Timestream, it’s helpful to start analyzing the data by visualizing it through dashboards and reports. Timestream provides native integrations with Amazon QuickSight and Amazon Managed Grafana, two powerful tools for building interactive dashboards, paginated reports, and embedded analytics. Additionally, a JDBC connector is available to allow you to use your business intelligence tools and other applications. These dashboards and reports are often necessary for compliance and auditing needs, as well as for monitoring infrastructure.

Dashboards in Timestream become even more powerful and cost-effective with the use of scheduled queries, a fully managed, serverless, and scalable solution for calculating and storing aggregates, rollups, and other real-time analytics. With scheduled queries, you simply define the queries on your incoming data, and Timestream periodically and automatically runs these queries and reliably writes the results into a configurable destination table. You can then point your dashboards, reports, applications, and monitoring systems to simply query the destination tables instead of querying the considerably larger source tables containing the incoming time series data. The destination tables contain much less data than the source tables, thereby offering faster and less expensive data access and storage.

Analyze data through the query editor and integrations

To generate real-time insights across your data, you can use the Timestream built-in query editor for running point lookups and analytical queries. The query editor is a powerful, SQL-friendly tool that caters to developers, data scientists, security analysts, and others.

Additionally, the native integration with AWS Lambda and the JDBC connector enable further analytics on your security data. These integrations make it simple for you to plug query results into your existing pipelines and use proprietary applications for various use cases. With the use of Lambda, your monitoring services can be deployed to analyze data collected in Timestream, while also alert on malicious signals.

You can further augment your analytics with prediction capabilities through Amazon SageMaker. SageMaker notebooks allow you to integrate your machine learning models with Timestream, and provide no-code visual interfaces for analysts and business intelligence teams.

Generate alerts on security incidents

Security analytics would not be complete without a quick and easy mechanism for notifying your team when action is required. With the integration in Timestream with Grafana, you can use Grafana alerts. Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results, and compare them to each other or to fix thresholds. When these are run, Grafana sends notifications to the contact point you specify. Grafana supports numerous contact points, including email, webhook, Microsoft Teams, PagerDuty, Slack, and more.

Additionally, you can use Amazon Simple Notification Service (Amazon SNS) through Lambda to send notifications. Amazon SNS provides high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications. With alerting, you can create real-time, automated alerts for monitoring security infrastructure and resolving incoming threats quickly.

Timestream is built for security

A robust security solution can help you detect and resolve threats before they become a problem. Timestream offers a number of capabilities in data ingestion, storage, analytics, and visualization to effectively and efficiently mitigate issues at scale. Also, Timestream has comprehensive security capabilities and is HIPAA eligible, ISO (9001, 27001, 27017, and 27018) certified, FedRAMP Moderate compliant, PCI DSS compliant, and in scope for AWS’s SOC reports SOC 1, SOC 2, and SOC 3.

Conclusion

In this post, we described how Timestream supports the core components of security analytics, and how VMware Carbon Black addresses their observability use case. To learn more about how VMware Carbon Black provides advanced security for native workloads hosted on AWS EC2, visit the VMware Carbon Black Workload User Guide. The VMware Carbon Black Workload vulnerability solution provides information on vulnerabilities that is available in Carbon Black Cloud Management Console.

To learn more and get started with Timestream, visit the Developer Guide, watch the Getting Started video, learn the best practices, and read about supported integrations.

About the Author

Igor Shvartser is a Senior Product Manager for Amazon Timestream. His fascination with data, working alongside customers, and building exceptional products has led him to AWS where he’s empowering teams with purpose-built databases.