DynamoDBTraceListener

We recently introduced the DynamoDBTraceListener, a System.Diagnostics TraceListener that can be used to log events straight to Amazon DynamoDB. In this post, we show how simple it is to configure the listener and how to customize the data that is being logged.

Configuration

You can configure the listener either through code or by using a config file. (For console applications, this will be app.config, while IIS projects will use web.config.) Here is a sample configuration that lists a few of the possible configuration parameters:

<system.diagnostics>
  <trace autoflush="true">
    <listeners>
      <add name="dynamo" type="Amazon.TraceListener.DynamoDBTraceListener, AWS.TraceListener"
                      Region="us-west-2"
                      ExcludeAttributes="Callstack"
                      HashKeyFormat="%ComputerName%-{EventType}-{ProcessId}"
                      RangeKeyFormat="{Time}"
        />
    </listeners>
  </trace>    
</system.diagnostics>

Web.config parameters

Here are all the possible parameters you can define in the config file, their meanings and defaults:

AWSAccessKey : Access key to use.
AWSSecretKey : Secret key to use. The access and secret keys can be set either in the listener definition or in the appSettings section. If running on an EC2 instance with a role, the listener can use the instance credentials. When specifying these, consider using an IAM user with a restricted policy like the example at the bottom of this post.
Region : Region to use DynamoDB in. The default is “us-west-2”.
Table : Table to log to. The default is “Logs”.
CreateIfNotExist : Controls whether the table will be auto created if it doesn’t exist. The default is true. If this flag is set to false and the table doesn’t exist, an exception is thrown.
ReadCapacityUnits : Read capacity units if the table is not yet created. The default is 1.
WriteCapacityUnits : Write capacity units if the table is not yet created. The default is 10.
HashKey : Name of the hash key if the table is not yet created. The default is “Origin”.
RangeKey : Name of the range key if the table is not yet created. The default is “Timestamp”.
MaxLength : Maximum length of any single attribute. The default is 10,000 characters (“10000”).
ExcludeAttributes : Comma-separated list of attributes that should not be logged. The default is null – all possible attributes are logged.
HashKeyFormat : Format of the hash-key for each logged item. Default format is “{Host}”. See format description below.
RangeKeyFormat : Format of the range-key for each logged item. Default format is “{Time}. See format description below.
WritePeriodMs : Frequency of writes to DynamoDB, in milliseconds. The listener will accumulate logs in a local file until this time has elapsed. The default is one minute (“60000”).
LogFilesDir : Directory to write temporary logs to. If you don’t specify a directory, the listener attempts to use the current directory, then the temporary directory. If neither is available for writing, the listener will be disabled.

Hash/range key formats

As you’ve noticed from our example, the hash and range keys can be compounded. The format can consist of strings, existing attribute names (e.g., {Host}), environment variables (e.g., %ComputerName%), or any combination of these. Here is an example that combines all possible approaches:

Prod-%ComputerName%-{EventType}

When constructing the format, you can use the following attributes: Callstack, EventId, EventType, Host, Message, ProcessId, Source, ThreadId, Time. These are also the attributes that can be excluded from being logged with the ExcludeAttributes configuration.

Using DynamoDBTraceListener programmatically

Should you need to create and use the listener in your code, this is a simple and straightforward operation. The next sample shows how a to create and invoke a listener.

DynamoDBTraceListener listener = new DynamoDBTraceListener
{
    Configuration = new DynamoDBTraceListener.Configs
    {
        AWSCredentials = new BasicAWSCredentials(accessKey, secretKey),
        Region = RegionEndpoint.USEast1,
        HashKeyFormat = "%ComputerName%-{EventType}"
    }
};
listener.WriteLine("This is a test", "Test Category");
listener.Flush();

Background logging

DynamoDBTraceListener logs events in two separate stages. First, we write the event data to a file on the disk. Then, at periodic intervals, these files are pushed to DynamoDB. We use this approach for a number of reasons, including asynchronous logging and the batching of writes, but most importantly it is done to prevent loss of data if the hosting application terminates unexpectedly. If this happens, we will push any existing log files the next time the application runs and the listener pushes logs to DynamoDB.

Even though the listener writes data to DynamoDB on a periodic basis, it is important to remember to flush the listener or to properly dispose of whatever resources you have that log, such as the client objects in the AWS SDK for .NET. Otherwise, you may find some of your logs are not being uploaded to DynamoDB.

When the listener first starts, we attempt to find a directory for the log files. Three different locations are considered: LogFilesDir, if one is configured by the user; the directory containing the current assembly; the current user’s temporary folder (as resolved by the Path.GetTempPath method). Once a location is determined, an information event is written to the Event Log specifying the current logging location. If none of these locations are available, however, an error event is written to the Event Log and the listener is disabled.

IAM user

For safety, you may not want to put your root account credentials in the application config. A much better approach is to create an IAM user with specific permissions. Below is an example of a policy that limits a user’s permissions to just DynamoDB and only for those operations that the listener actually uses. Furthermore, we’re limiting access to just the log table.

{
  "Statement" : [
    {
      "Effect" : "Allow",
      "Action" : [
        "dynamodb:DescribeTable",
        "dynamodb:CreateTable",
        "dynamodb:BatchWriteItem"
      ],
      "Resource" : "arn:aws:dynamodb:us-west-2:YOUR-ACCOUNT-ID:table/Logs"
    }
  ]
}

IAM Roles

If you are using DynamoDBTraceListener in an environment that is configured with an IAM Role, you can omit the AWSAccessKey and AWSSecretKey parameters from the config file. In this case, DynamoDBTraceListener will access DynamoDB with permissions configured for the IAM Role.

AWS Developer Tools Blog