How to Analyze Fastly Content Delivery Network Logs with Amazon QuickSight Powered by Generative BI

By Vladimir Vuksan, Sr. Principal Sales Engineer – Fastly
By Sindhura Palakodety, Sr. Solutions Architect – AWS
By Manish Mishra, Sr. Technical Account Manager – AWS

Fastly

A content delivery network (CDN) improves efficiency by caching content in servers closer to users, reducing the time needed to send data-heavy web content and speeding up webpage loading.

CDNs also provide global reach, cost savings, and enhanced reliability through automatic redundancy between edge servers, as well as improved security and protection against distributed denial-of-service (DDoS) attacks.

CDN performance has a direct impact on user experience. A robust CDN monitoring strategy is essential for maintaining optimal performance, improving mean time to resolution and for making informed business decisions.

Analyzing CDN logs provides business analysts and website administrators with the ability to understand traffic patterns and react by taking necessary actions to optimize content delivery, especially during peak traffic periods.

Fastly is an AWS Specialization Partner and AWS Marketplace Seller that’s a CDN provider helping to deliver content faster and efficiently. Logs from Fastly CDN can be quickly and easily understood by landing them in Amazon Simple Storage Service (Amazon S3).

With Amazon QuickSight powered by generative business intelligence (BI), you can make sense of the data by easily building visualizations, performing ad-hoc analysis, and quickly gaining business insights from your data.

This post demonstrates how you can quickly build an Amazon QuickSight dashboard to gain visibility and insights into the logs being generated by Fastly CDN. We’ll also show examples of how business analysts can utilize the new generative BI capabilities in QuickSight for:

• Accelerating the dashboard creation process by describing the visuals they want in natural language.
• Editing visuals by describing changes using natural language.
• Creating and sharing rich data stories with executives.

Solution Overview

In this post, we’ll configure a Fastly real-time log streaming using Amazon S3, populate data catalog using AWS Glue, and visualize the data using Amazon QuickSight. Note that each of these services will incur relevant charges. Refer to the pricing documentation for each of these services before using them.

• Amazon S3 stores the CDN logs coming from Fastly.
• AWS Glue crawlers catalog the data.
• Amazon Athena queries the processed data.
• Amazon QuickSight dashboards provide insights into various CDN metrics.

Refer to the Fastly CDN QuickSight dashboard with sample data available on the community arena.

Figure 1 – Fastly CDN QuickSight dashboard.

The following diagram illustrates the solution architecture. Fastly CDN logs are stored in an Amazon S3 bucket periodically, based on the interval configured in the CDN settings. AWS Glue crawler can be triggered on the S3 bucket for updating the Glue Data Catalog, which can then be queried using Amazon Athena.

Amazon QuickSight can be used to visualize the logs from the Athena dataset and can provide us with valuable insights.

Figure 2 – High-level architecture.

Configure Your Fastly Delivery Service

• Create a Fastly account if you don’t have one.
• From the Fastly home page, select Create a delivery service. Refer to Fastly documentation.
• Add the domain name of your website in the Domain section.
• Select Origins > Host to add your host name/IPV4 address.
• Log in to the AWS console and create an Amazon S3 bucket which will used for storing CDN logs from Fastly.
• Select Logging and follow the documentation to set up your S3 logging. The S3 bucket needs to be the one that was created in the prerequisites above. Note the default Period for log file rotation is 3,600 seconds, which means that logs will be populated in the S3 bucket every hour. Be sure to configure it based on your needs.
• Name your created delivery service (Options > Edit your Service name) as per your choice.
• Check the S3 bucket to verify if the log files are being generated. The log files may show up periodically depending upon the value of log Period configured above.

AWS CloudFormation Prerequisites

• The AWS CloudFormation template linked to below requires access to Amazon Glue services (crawler, database, and tables), the Amazon S3 bucket where the CDN logs are stored, and Amazon QuickSight. Hence, ensure the user/role which invokes the CloudFormation stack has relevant permissions.
• Provide QuickSight access to Amazon Athena and S3 services:
  • Log in to the QuickSight console.
  • Choose your profile name located in the upper right corner.
  • Choose Manage QuickSight, then Security & permissions, and finally Manage.
  • Select Amazon Athena.
  • Select Amazon S3 in the list and choose the S3 bucket with the CDN logs.
  • Click on Save.
• You’ll also need a QuickSight user with appropriate permissions to create QuickSight datasource, dataset, and analysis is needed. Note down the Amazon Resource Name (ARN) of this user in this format: arn:aws:quicksight:<region>:<account-id>:user/default/<IAM user>
  • For example: arn:aws:quicksight:us-east-1:123456789012:user/default/johndoe

Launch CloudFormation Stack

To launch the provided CloudFormation template, complete the following steps:

• Choose Launch Stack below to launch the CloudFormation stack.

• Ensure you use the AWS region where the CDN logs S3 bucket is located.
• Choose Next.
• For Stack name, enter a name.
• Provide the following parameters:
  • Name of the S3 bucket that stores Fastly CDN logs.
  • ARN of QuickSight user who has required access to create QuickSight datasource, datasets and analysis.
• Enter any tags you want to assign to the stack, and choose Next.
• Select the acknowledgement check boxes, and choose Create stack.

The stack takes approximately 10 minutes to complete. On the CloudFormation console, you can navigate to the stack’s Resources tab to review the resources you created. The CloudFormation stack creates a Glue crawler, database, and table, as well as a QuickSight datasource, dataset, and analysis. Note the CloudFormation stack will set up the Glue crawler with an “on demand” frequency. If you’d like to schedule it on a recurring basis based on your need, refer to the documentation.

Leverage Generative BI Capabilities in QuickSight

• Sign up for QuickSight Enterprise Edition.
• Include the Amazon Q add-on for your QuickSight account.
• Enable Generative BI with Amazon Q in QuickSight.
• Create a QuickSight Topic from the CDN logs QuickSight dataset. If you have existing Amazon Q topics related to CDN logs, you can easily convert these to leverage the new generative capabilities using the steps in the documentation.

Refer to the QuickSight pricing documentation for more details on the pricing associated with the capabilities above.

Creating QuickSight Dashboard from Analysis

You can publish the QuickSight analysis into a dashboard:

• Head to the QuickSight console. Note that you’ll need to use the same region where the CloudFormation stack is deployed.
• Verify whether your QuickSight dataset and analysis have been created successfully by CloudFormation.
• Create a QuickSight dashboard from the analysis you created. For instructions, refer to Tutorial: Create an Amazon QuickSight dashboard.

While publishing the dashboard, if you’d like to leverage generative BI capabilities you can associate the QuickSight topic that was created above with it.

Powering Dashboards with Generative BI

Amazon QuickSight authors can use the Build a visual button to build a custom visual that’s generated from your input. You can enter a custom description, or you can choose from a list of generated suggestions that Amazon Q has generated for the topic that’s attached to the analysis.

The following image shows a custom visual that’s created with the Build a visual menu. For example, if you’re interested in finding out access patterns of a specific client IP 73.22.87.7 with time, you can click on Build a Visual option at the top of the QuickSight analysis and enter a description in natural language.

Figure 3 – Building custom visual using natural language via QuickSight analysis.

Similarly, it’s possible to create visualizations in the QuickSight dashboard. You can ask questions with Amazon QuickSight Q by choosing the Ask a question option in the dashboard. This option also comes up with suggestions regarding questions you can ask and get appropriate multi-answer visualizations.

Figure 4 – Ask a question to Amazon QuickSight Q.

For example, the following visualizations highlights the fact that GET request method accounts for the majority of requests. You can add these visualizations to your PINBOARD.

Figure 5 – Multi-answer visualizations using Amazon QuickSight Q.

You can also leverage Named Entities if you’d like to present contextual multi-visual answers. Named entities are one of the most important components of topic curation. The information contained in named entities—specifically, the ordering of fields and their ranking makes it possible to present such answers.

For example, if you’d like to deep dive into responses associated with GET requests, you can configure the Named Entity and receive multi-visual answers.

Figure 6 – Using named entities.

Edit Existing Visuals Using Generative BI

Amazon QuickSight authors can also use natural language prompts to edit visuals in an analysis, as shown in the following visual. Authors can use this functionality to edit visuals without performing manual tasks in the QuickSight user interface (UI).

For example, to highlight the color of the client_ip field values where number of requests are more than 50, you can simply state the following:

Figure 7 – Edit visuals using generative BI.

Creating a Data Story with Amazon Q in QuickSight

Follow the steps in the documentation to create a data story in Amazon QuickSight about your website performance and share it with your executives.

Figure 8 – Creating a data story.

Cleaning Up

To avoid incurring future charges, delete the resources you created as part of this solution.

Conclusion

Congratulations! You have successfully built a generative BI dashboard on Amazon QuickSight which provides in-depth visualizations into Fastly CDN metrics. With the steps and resources described in this post, you can build your own analytics dashboard leveraging generative BI QuickSight capabilities.

.

.

Fastly – AWS Partner Spotlight

Fastly is an AWS Specialization Partner that helps businesses make great digital experiences happen quickly, securely, and reliably by processing and serving customers’ applications at the edge, as close to end-users as possible.

Contact Fastly | Partner Overview | AWS Marketplace