AWS Partner Network (APN) Blog
Scaling AWS multi-region and account logs delivery to Grafana Cloud
By Vasil Kaftandzhiev, Senior Product Manager, Grafana Labs
By Anuj Sharma, Principal Solutions Architect, AWS
 |
| Grafana Labs |
 |
In today’s cloud-centric world, organizations often find themselves managing multiple AWS accounts across various regions. As the complexity of these environments grows, so does the challenge of effectively monitoring and analyzing logs from disparate sources. Centralized log monitoring is no longer a luxury but a necessity for maintaining visibility, ensuring compliance, and optimizing performance. This blog post explores the benefits of creating a scalable log delivery architecture using AWS CloudWatch as the source and Grafana Cloud as the destination.
The Need for Centralized Log Monitoring
Customers may adopt multiple AWS regions and AWS accounts to run workloads for reasons such as redundancy, higher availability, disaster recovery, data sovereignty, compliance, latency optimization, security and optimization. Distributed cloud environments generate a volume of logs, making it increasingly difficult to gain actionable insights from siloed data sources. Customers often need to forward all or a subset of logs generated by AWS services such as Amazon Bedrock, AWS Lambda, Amazon Elastic Kubernetes Services (EKS). Centralized log monitoring empowers organizations to consolidate logs from multiple AWS accounts and regions, enabling comprehensive analysis, correlation, and troubleshooting. By streamlining log management, businesses can proactively identify and resolve issues, enhance security posture, and ensure regulatory compliance.
Architecture Overview
The proposed architecture leverages AWS CloudWatch account level subscription. With account-level subscription filters, customers can egress logs ingested into multiple or all log groups by setting up a single subscription filter policy for the entire account. This saves time and reduces management overhead. The account-level subscription filter applies to both existing log groups and any future log groups that match the configuration. Each account can create one account-level subscription filter. Using Amazon Data Firehose, logs can be ingested in Grafana Cloud, which then serves as the destination and provides a unified interface for log analysis, visualizations, and alerting.. This architecture ensures scalability, reliability, and cost-effectiveness, enabling organizations to monitor their AWS environments with ease.
Figure-1 – Use CloudWatch account level subscription filter to send logs from multiple AWS Accounts to a single data Firehose
The architecture of this solution includes the following steps:
- In a specific AWS account, where you run your workloads using AWS services such as AWS Lambda, Amazon Bedrock etc., the logs are sent to AWS CloudWatch logs in a log groups.
- You then create Amazon Data Firehose (ADF) with Grafana Cloud’s endpoint as destination and the API Key. Typically, you would create ADF in a centralized AWS account, meant to egress the data to Grafana Cloud. You would also create a role, which has the access to put data to ADF with a trusted principal of logs.amazonaws.com. This IAM role needs to have a policy attached that includes a aws:SourceArn global condition context key that corresponds the source AWS account ID to help prevent the confused deputy security problem and action has sts:AssumeRole .
- You then create a log destination in the centralized monitoring account for each AWS region where your workloads will send data. Each log destination is configured to use the IAM role thus created. Once you create a log destination, you would create a policy attached to the destination, that allows logs:PutSubscriptionFilter and logs:PutAccountPolicy from the source AWS account
- In the source AWS account, you then create an account level subscription filter policy (per AWS region) to send logs from a subset of AWS CloudWatch log groups or all log groups to a log destination in the destination account. The sending account’s log groups and the destination must be in the same AWS Region. However, the destination can point to ADF that is located in a different Region.
- Now, logs from any log group (thus defined in the account level subscription filter policy), existing or new, will get sent to the CloudWatch log destination in the centralized monitoring AWS account and from there to Grafana Cloud via ADF.
Benefits of Centralized Log Monitoring with AWS CloudWatch and Grafana Cloud
By implementing this architecture, organizations can unlock numerous benefits, including:
- Improved visibility and operational efficiency through centralized log analysis and correlation.
- Scale security and compliance by consolidating logs from multiple sources for comprehensive auditing and threat detection.
- Streamlined troubleshooting and root cause analysis with advanced log search and filtering capabilities.
- Scalable and cost-effective log management, leveraging the power of CloudWatch and Grafana Cloud’s robust infrastructure.
- Customizable dashboards and alerting mechanisms for proactive monitoring and incident response.
Conclusion and Call to Action
In the ever-evolving cloud landscape, centralized log monitoring is a critical component of effective cloud operations. By combining the power of CloudWatch and Grafana Cloud, organizations can gain comprehensive visibility into their AWS environments, streamline log management, and unlock valuable insights for optimized performance, security, and compliance.
To learn more about implementing this scalable log delivery architecture, visit Grafana Labs’ step-by-step guide and documentation or follow the self-guided onboarding process directly from the Grafana Cloud console. Unlock the full potential of your AWS investments and take control of your log monitoring strategy today by subscribing to Grafana Cloud from AWS Marketplace.
.
.
Grafana Labs – AWS Partner Spotlight
Grafana Labs is an AWS Advanced Technology Partner and AWS Competency Partner that provides Grafana Cloud delivering the LGTM stack as a service in addition to on call, alerting, K6 and all other Grafana platform features.