AWS Fargate container logs collection and analysis with AWS FireLens and Sumo Logic

AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters. Fargate manages provisioning, configuration, and scaling of the clusters. With Fargate, you can focus on your application design and implementation instead of worrying about the infrastructure.

In this post, we’ll provide an overview on how to retrieve Fargate container logs and send those to Sumo Logic using AWS FireLens to easily monitor and troubleshoot container and application problems.

AWS Firelens for Amazon ECS enables you to use task definition parameters to route logs to an AWS service or AWS Partner Network (APN) destination for log storage and analytics. FireLens works with open source technologies Fluentd and Fluent Bit.

Fluentd is an open source data collector for the unified logging layer. Fluentd allows you to unify data collection and consumption for better use and understanding of data. Fluentd is written in a combination of C language and Ruby, and requires very few system resources: The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core.

Fluent Bit is the lightweight forwarder for Fluentd. It includes multiple Fluentd plugins that parse and format the metrics and enrich them with metadata. Data is enriched — tagged — with details about where in the cluster it originated, the service, deployment, namespace, node, pod, container, and their labels. It then forwards logs and metrics to an HTTP source on a hosted collector. Fluent Bit is an open source and multi-platform log processor and forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It’s fully compatible with Docker and Kubernetes environments.

Sumo Logic is an AWS Partner Network (APN) Advanced Technology Partner with AWS competencies in Security, Data and Analytics, and DevOps. Sumo Logic helps organizations gain better real-time visibility into their IT infrastructure. Sumo Logic integrates with many cloud  as well as on-prem services, making it simple and easy to aggregate data across different services and giving users a full view of their operational, business, and security analytics.

AWS Fargate logs collection setup

A task definition is required to run Docker containers in Amazon ECS. You can define multiple containers in a task definition. The parameters that you use for the task definition depend on the launch type for the task. For more information about available parameters and the launch types they are valid for in a task definition, see the ECS task definition parameters documentation.

Amazon ECS allows two launch types, Fargate and EC2.

●      Fargate: ECS manages your clusters.
●      EC2: you manage your clusters.

With AWS FireLens, you can configure the open source collectors Fluentd or Fluent Bit in your ECS task definitions for both AWS Fargate and Amazon EC2. Of the two collector agents, Fluent Bit is recommended, because its resource utilization is significantly lower than Fluentd’s. The following diagram illustrates how FireLens sends container logs from Fargate to Sumo Logic.

Figure 1: AWS Fargate log ingestion into Sumo Logic using AWS Firelens

The integration configuration steps are as follows:

Configure a hosted collector and HTTP logs source in Sumo Logic

A hosted collector is not installed on a local system in your deployment. Instead, Sumo Logic hosts the collector and its sources in AWS. With a hosted collector, you can create sources to collect data from various services. A single hosted collector can be configured with any number of  sources.

An HTTP logs and metrics source is an endpoint for receiving log and metric data uploaded to a unique URL generated for the source. The URL securely encodes the collector and source information. You can add as many HTTP logs and metrics sources as you’d like to a single hosted collector.

You’ll obtain an HTTP URL for the source containing an endpoint and a token which will be used in the next steps.

Create a FireLens log router container definition with either Fluentd or Fluent Bit, and mark it as the FireLens container

{
    "image": "amazon/aws-for-fluent-bit:latest",
    "name": "log_router_sumo",
    "memory": 100,
    "essential": true,
    "firelensConfiguration": {
      "type": "fluentbit",
      "options": {
         "enable-ecs-log-metadata": "true"
       }
     }
}

Create application containers that use AWS FireLens for logs and configure the Sumo Logic HTTP source endpoint and token to send the logs to Sumo Logic

{
  "essential": true,
  "image": "httpd",
  "name": "app",
  "logConfiguration": {
    "logDriver": "awsfirelens",
    "options": {
      "Name": "http",
      "Host": "<endpoint>",
      "URI": "/receiver/v1/http/<token>",
      "Port": "443",
      "tls": "on",
      "tls.verify": "off",
      "Format": "json_lines"
     }
   },
   "memory": 100
}

Now that your container logs are configured to be sent to Sumo Logic, verify that they are indeed being collected by following the instructions in How can I tell if I’m collecting data?

For more detailed instructions, refer to Sumo Logic’s instructions to Collect AWS ECS Fargate and EC2 Container Logs.

Searching container logs in Sumo Logic for troubleshooting

Once log collection is configured, the logs will start flowing into Sumo Logic. Below is an example of an Apache webserver log:

Figure 2: Apache web logs ingestion into Sumo Logic

FireLens adds ECS metadata which is very useful for investigations. Using searches you can easily determine the errors, performance, and health of the containers.

Leveraging app dashboards

Based on the applications running in your Fargate container, you can automatically get insights into the data using Sumo Logic applications.

In the above example, since Apache logs are being ingested into Sumo Logic, the Sumo Logic Apache app can be installed to analyze logs and trends via out-of-the-box dashboards.

The Apache – Overview dashboard in this app provides an at-a-glance view of visitor locations and traffic distribution:

Figure 3: Visitor locations and traffic distribution in Apache

For more details on how to setup the Apache app and other dashboards in this app, please see the instructions to install the Sumo Logic Apache application.

Cleanup

To avoid incurring future charges to your AWS accounts, delete the resources created in your AWS account for this project. You can simply delete the Fargate cluster and delete the Sumo Logic free trial you created by going to Administration > Account in the Sumo Logic web page. At the bottom of the page, under the section Delete this Organization from Sumo Logic?, select Delete Org. In the dialog Delete This Organization from Sumo Logic?, enter DELETE to confirm and click Delete Org.

Summary

In this post, we have shown an overview on how the Sumo Logic integration with AWS ECS works so you can:

• Collect container logs from AWS Fargate using AWS Firelens with open source technologies Fluentd and Fluent Bit.
• Troubleshoot and investigate issues using Sumo Logic searches.
• Leverage out-of-the-box Sumo Logic applications to get instant insights.

Next steps

To get started, check out the Sumo Logic ECS Log collection help doc. If you don’t yet have a Sumo Logic account, you can sign up for a free trial.

For more security and DevSecOps-focused reads, check out the Sumo Logic blog.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.