Alternative Formatting for Metrics Data in .NET SDK Logs

The AWS SDK for .NET has had response logging and performance metrics logging since before version 2.0. We introduced SDK logging and metrics output in an earlier post. You might want to skim that as a refresher.

The metrics data is included in the logs in a human-readable format, but SDK users who aggregate, analyze, and report on this data have to implement their own parser to parse the data out of the logs, which takes time, and can be error prone. So, we’ve added an alternate format to emit the metrics data into the log as JSON.

If you need a format other than JSON, or if you only need to log a subset of the metrics, the SDK now also has a mechanism to add a custom formatter.

Switching to JSON is done through the App.config or Web.config. The SDK’s application configuration has changed a little bit since the aforementioned post; though, for the sake of backwards compatibility, the original configuration settings will still work. To use the JSON setting, however, you’ll have to adopt the new style of configuration, at least for the logging section.

The old style of configuration was a set of flat key-value pairs in the <appSettings> section, like this:

<appSettings>
    <add key="AWSLogging" value="SystemDiagnostics" />
    <add key="AWSLogMetrics" value="true" />
    <add key="AWSResponseLogging" value="OnError" />
</appSettings>

The new configuration uses a custom section for the SDK with a structured format, like this:

<configuration>
    <configSections>
        <section name="aws" type="Amazon.AWSSection, AWSSDK" />
    </configSections>
    ...
    <aws region="us-west-2">
        <logging logTo="SystemDiagnostics"
             logMetrics="true"
             logResponses="OnError"
             logMetricsFormat="JSON" />
    </aws>
    ...
</configuration>

You can see that this configuration selects the JSON formatting. The rest of the logging configuration, including selection of System.Diagnostics or Log4Net, is the same as specified in the introductory logging post.

Creating a custom formatter is easy too. First, you need to implement the Amazon.Runtime.IMetricsFormatter interface, specifying a single method that takes in an Amazon.Runtime.IRequestMetrics and returns a string. Here’s a trivial example that prints out a single metric for a request:

using Amazon.Runtime;
namespace MyLib.Util
{
    public class MyMetricsFormatter : IMetricsFormatter
    {
        public string FormatMetrics(IRequestMetrics metrics)
        {
            var fmt = string.Empty;
            if (metrics.Timings.ContainsKey(Metric.ResponseProcessingTime))
            {
                var timing = metrics.Timings[Metric.ResponseProcessingTime]
                    .FirstOrDefault();

                if (timing != null)
                    fmt = string.Format("ResponseProcessingTime (ms): {0}", 
                            timing.ElapsedTime.Milliseconds);
            }
            return fmt;
        }
    }
}

The IRequestMetrics has three dictionaries of metrics; Properties, Timings, and Counters. The keys for these dictionaries are defined in the Amazon.Runtime.Metric enum. The Properties and Timings dictionaries have lists as values, and the the Counters dictionary has long as values.

To use a custom formatter, use the logMetricsCustomFormatter configuration, specifying the type and assembly:

<aws region="us-west-2">
    <logging logTo="SystemDiagnostics"
         logMetrics="true"
         logResponses="OnError"
         logMetricsCustomFormater="MyLib.Util.MyMetricsFormatter, MyLib" />
</aws>

If you want to collect metrics for a subset of services or method calls, your custom formatter can inspect the Metrics.ServiceName and Metrics.MethodName items in the Properties dictionary. The default behavior can be accessed by calling ToString() on the passed in IRequestMetrics. Similarly, you can get the JSON by calling metrics.ToJSON().

Keep in mind that if you have metrics logging enabled and have specified a custom formatter your formatter will be called for every request, so keep it as simple as possible.