Microsoft Workloads on AWS

.NET observability with Amazon CloudWatch and AWS X-Ray: Part 2 — Logging

Building a well-architected .NET application goes beyond just coding and deploying. You must monitor performance, trace transactions, collect logs, gather metrics, and trigger alarms when metrics breach thresholds. To achieve this, you can design and implement telemetry to enable observability capabilities.

In the first part of this blog series, I covered the implementation of metrics. In this second post, I will cover logging implementation on .NET Applications using .NET C# logging with Amazon CloudWatch Logs, and then I’ll show you how to correlate these logs with metrics and distributed traces using Amazon CloudWatch and AWS X-Ray.

Logging is a key observability capability. It lets you capture data points at runtime so that you can filter, transmit, and store them for further analysis or troubleshooting purposes. Logging also helps you debug errors, monitor performance, track user behavior, trace transactions, and more.

Solution overview

Consider a containerized .NET workload, like the one in Figure 1, which runs on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. It comprises three .NET microservices: an ASP.NET Core web API, two Worker Services, plus the cloud resources they use. A system or platform in an actual production scenario could have dozens, hundreds, or even thousands of these microservices. If you had to troubleshoot multiple microservices, it could be time-consuming or nearly impossible if you didn’t have logging implemented in each microservice.

To mitigate this scenario, you can emit logs from the key steps in each component to include relevant data points that help you understand what is occurring during processing. But if the logs are all isolated from each other, they may not help you get to the root cause quickly. So processing them in a centralized system that can correlate them with other data points, such as metrics and distributed traces, will help.

Although it could be challenging to implement all components of observability and process them centrally, Amazon CloudWatch and AWS X-Ray can help you address this challenge. In this post, I will show you how to implement .NET C# logging with CloudWatch Logs and instrument the logs with AWS X-Ray Trace information so that Amazon CloudWatch can correlate the logs with other observability components.

Architecture Diagram

Figure 1: .NET Microservice solution

Prerequisites

For this example, you can use the GitHub repository “microservices-dotnet-aws-cdk,” which contains the sample code for all three microservices. It uses the AWS Cloud Development Kit (CDK) to define and provision the Infrastructure as code using the C# programming language.

The following prerequisites are required on your system to test and deploy this solution:

You can clone the repository from the command line with the following command:

git clone https://github.com/aws-samples/microservices-dotnet-aws-cdk.git

Open the solution in an IDE, such as Visual Studio Code, to explore the implementation.

Implement logging with CloudWatch Logs on .NET application

.NET C# supports a Logging API specification with various built-in and third-party logging providers. A best practice for better filtering and correlation is to use a provider that generates JSON outputs. The JSON format helps with parsing, filtering, and visualization.

To improve the troubleshooting experience, you can add the AWS X-Ray trace ID in every log output. Then, CloudWatch can correlate all data points with that same Trace ID. These data points can be metrics, logs, and distributed traces. Including the trace ID in each log output helps simplify the troubleshooting process, allowing you to filter and focus on the logs that have the same trace ID from different components of your workload. It also allows you to expand the view to show individual metrics and distributed traces to get the insight to needed to help with troubleshooting.

The following code sample demonstrates a Logging implementation using the built-in C# provider in a one of the Worker Services applications to add the Trace ID in each log:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    ...
    while (!stoppingToken.IsCancellationRequested)
    {
        ...
        try
        {
            var messageId = await ReceiveAndDeleteMessage(_sqsClient, queueUrl);
            _logger.LogInformation("Message ID: {messageId}, TraceId: {TraceId}", messageId, traceEntity.TraceId);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "error consuming SQS Queue");
            AWSXRayRecorder.Instance.AddException(ex);
        }
        finally
        {
            var traceEntity = AWSXRayRecorder.Instance.TraceContext.GetEntity();
            AWSXRayRecorder.Instance.EndSegment();
            AWSXRayRecorder.Instance.Emitter.Send(traceEntity);
            _logger.LogDebug("Trace sent {TraceId}", traceEntity.TraceId);
        }
       ...
    }
}

To avoid repeating the same portion of code in each logging call, .NET allows the implementation of a custom logging provider or formatter in which you add the trace ID logic once, and then it’s used every time an application code invokes logging methods. The following is example code that implements a custom logging formatter:

public class XrayCustomFormatter : ConsoleFormatter, IDisposable
{
    ...
    public override void Write<TState>(
        in LogEntry<TState> logEntry,
        IExternalScopeProvider scopeProvider,
        TextWriter textWriter)
    {
        ...
        WriteTraceIdSuffix(textWriter);
        ...
    }

    private void WriteTraceIdSuffix(TextWriter textWriter)
    {
        if (_formatterOptions.EnableTraceIdInjection && AWSXRayRecorder.Instance.IsEntityPresent())
        {
            textWriter.Write($"TraceId: {AWSXRayRecorder.Instance?.GetEntity()?.TraceId}");
        }
    }
    ...
}

Send the logs to CloudWatch Logs

You should use a separate process to handle the transmission of the log to the CloudWatch Logs so the log processing logic doesn’t affect application performance. The log transmission mechanism will vary, depending on which AWS services you are using to host your .NET application. Here are some options:

Since I am using Amazon ECS and AWS Fargate for this sample application, the following code sample shows how to provision the Amazon ECS log driver using AWS CDK. AWS CDK lets you build applications in the cloud as code, with the benefit of the expressive power of programming languages, such as C#, which I am using here:

//CloudWatch LogGroup and ECS LogDriver
var logGroupName = "/ecs/demo/ecs-fargate-dotnet-microservices";
var logDriver = LogDriver.AwsLogs(new AwsLogDriverProps
{
    LogGroup = new LogGroup(this, "demo-log-group", new LogGroupProps
    {
        LogGroupName = logGroupName,
        Retention = RetentionDays.ONE_DAY,
        RemovalPolicy = cleanUpRemovePolicy
    }),
    StreamPrefix = "ecs/web-api"
});

 var albFargateSvc = new ApplicationLoadBalancedFargateService(this, "demo-service", new ApplicationLoadBalancedFargateServiceProps
{
    ...
    TaskImageOptions = new ApplicationLoadBalancedTaskImageOptions
    {
       ...
        LogDriver = logDriver,
    },
});

Deploy the example solution

The repository contains the full implementation of this solution, allowing you to make an HTTP request to the sample Web API to test it. To deploy, run one of the following deployment scripts in your environment using either bash (Linux or Mac) or PowerShell to deploy the solution.

Using bash:

./deploy.sh

Using PowerShell:

.\deploy.ps1

After deploying, copy the URL printed by the deployment script. It has the following format: http://WebAp-demos-XXXXXXXX-99999999.us-west-2.elb.amazonaws.com/api/Books. The Xs and 9s will be alphanumeric characters representing your deployment’s unique ID. Then, using a REST API client (such as Thunder Client for VS Code), test the solution by submitting an HTTP POST request to the URL with the following JSON payload. When you submit the HTTP POST, you should receive a response of status 200, and the TraceId result. Copy the TraceId value for later use.

{
    "Year" : 2022,
    "Title": "Demo book payload",
    "ISBN": 12345612,
    "Authors": ["Author1", "Author2"],
    "CoverPage": "picture1.jpg"
}

Figure 2 illustrates the example API call.
Example API call

Figure 2: Example API call

Visualizing the results

After making a test request to the API, you can navigate to Amazon CloudWatch to review the generated Logs.

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose X-Ray Trace, Traces.
  3. Leave the Filter by X-Ray Group Enter the trace ID you previously saved in the next field, and choose Run query.

The console will present a page similar to Figure 3. CloudWatch correlates all the data points from different microservices with the trace ID and displays them on this page. The page displays:

  • A Distributed Trace map.
  • Metrics for segments with a timeline of all microservices and components that processed the request, including the duration and status for each.
  • The logs for the services associated with the trace ID. You can analyze the data with more advanced queries on choosing View in the CloudWatch Logs Insights.

Logs filtered by trace IDFigure 3: Logs filtered by trace ID

Cleanup

You should clean up the resources created by running this demo to avoid unexpected charges. To do so, run the script from the root folder where you’ve cloned the GitHub repository.

Using bash:

./clean.sh

Using PowerShell:

.\clean.ps1

Conclusion

In the second post in this blog post series on implementing observability for your .NET applications on AWS, I’ve covered how to implement logging using. NET C# logging with CloudWatch Logs, and I’ve demonstrated how you can use AWS X-Ray trace ID to enable Amazon CloudWatch and AWS X-Ray to simplify troubleshooting and visualizing your application’s health. To learn about Metrics or Distributed Tracing implementations, read the other two parts of this series, “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 1 – Metrics” and “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 3 – Distributed Trace”.

Observability goes beyond instrumenting your .NET application to generate traces, logs, and metrics. You can leverage CloudWatch capabilities for monitoring, alarms, detecting anomalies, and more. See the Amazon CloudWatch Features page to learn more, and for hands-on experience, check out the One Observability Workshop.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your migration journey today.

TAGS: , ,
Ulili Nhaga

Ulili Nhaga

Ulili Nhaga is a Cloud Application Architect at Amazon Web Services in San Diego, California. He helps customers migrate, modernize, architect, and build highly scalable cloud-native applications on AWS. Outside of work, Ulili loves playing soccer, running, cycling, Brazilian BBQ, and enjoying time on the beach.