Ryan Green @ryangtweets
Software Development Engineer, API Gateway

Developing, testing, and operating Serverless APIs using Amazon API Gateway and AWS Lambda can be made much easier with built-in support for Amazon CloudWatch Logs.

In Lambda functions, you can use log statements to send log events to CloudWatch log streams, and API Gateway automatically submits log events for requests to APIs with logging enabled.

However, it can be difficult to reconcile log events for a serverless API sent across multiple CloudWatch log groups and log streams. Tracking down logs for a specific request or tailing request logs for that operation can sometimes be a cumbersome experience.

Here are some simple logging techniques and tools that can help greatly while developing, testing, and operating your serverless API.

Technique: Capture the request ID for a particular API request

The request ID for an individual API request is always returned by API Gateway in the ” x-amzn-RequestId” response header. When API Gateway logs an individual API request, the request ID is always included in the first event in the request log:

“Starting execution for request: [REQUEST_ID]”

Logs for an API Gateway API are always sent to a log group in the following format:

“API-Gateway-Execution-Logs_[API_ID]/[STAGE_NAME]”

To troubleshoot an individual API request, search for the request ID in the CloudWatch Logs console, or using the Cloudwatch API or an AWS SDK (more on tooling later).

Technique: Correlate your API Gateway request IDs with Lambda request IDs

Individual API requests are tracked independently across AWS services. Thus, an individual request to your API actually generates at least two request identifiers (“request IDs”) – one for the API Gateway request and one for the Lambda invocation request.

To reduce time-to-diagnosis when analyzing logs, it is helpful to correlate both request IDs together in the logs. Log the API Gateway request ID from your Lambda function and send the API Gateway request ID ($context.requestId) to your Lambda function via a mapping template:

{
   "context" : {
      "request-id" : "$context.requestId"
   }
   …
}

Then, in your Lambda function, log the API Gateway request ID along with the Lambda request ID. The Lambda request ID is automatically included in the log message.

exports.handler = function(event, context) {
    var apiRequestId = event.context['request-id'];
    var lambdaRequestId = context.awsRequestId;
    console.log("API Gateway Request ID: " + apiRequestId + " Lambda Request ID: " + context.awsRequestId);
    var logprefix = "APIG: " + apiRequestId + " -  ";
    console.log(logprefix + "hello world!");
    ...
}

Invoking this Lambda function produces these log messages in the “/aws/lambda/[FUNCTION_NAME]” log group:

/aws/lambda/echo                           2016/09/07/[$LATEST]6ccf17d298b64b5fac8c41b1a65e0831 2016-09-07T21:39:39.145Z        943ad105-7543-11e6-a9ac-65e093327849        API Gateway Request ID: 9439989f-7543-11e6-8dda-150c09a55dc2 Lambda Request ID: 943ad105-7543-11e6-a9ac-65e093327849

/aws/lambda/echo                           2016/09/07/[$LATEST]6ccf17d298b64b5fac8c41b1a65e0831 2016-09-07T21:39:39.145Z        943ad105-7543-11e6-a9ac-65e093327849        APIG: 9439989f-7543-11e6-8dda-150c09a55dc2 -   hello world!

Using this technique allows you to quickly locate the Lambda function logs for an individual API request. Search for the request ID returned in the “x-amzn-RequestId” header in the log group for the Lambda function (by default, named “/aws/lambda/[FUNCTION_NAME]”).

Tooling

While the AWS SDK and CLI provide excellent building blocks for working with API Gateway, Lambda, and CloudWatch Logs, there is still room for specialized tooling for logs when developing, testing, and operating your serverless API.

To that end, I’ve created apilogs—a fork of the excellent awslogs project—to include native support for API Gateway/Lambda serverless APIs.

Given an API Gateway REST API ID and Stage name, this command-line tool produces an aggregated stream of time-ordered, ANSI-colored log events emitted by API Gateway and all Lambda functions attached to your API. It automatically aggregates events from all log streams for the API and Lambda functions.

For example:

Stream all log events emitted from API Gateway as well as from all Lambda functions attached to the API:

$ apilogs get --api-id xyz123 --stage prod –-watch

Search APIG/Lambda logs for events from a specific request ID in the past hour:

$ apilogs get --api-id xyz123 --stage prod --start='1h ago' | grep "6605b081-6f04-11e6-97ac-c34deb0b3dd9"

The log events can then be further filtered and processed by standard command-line tools. Credentials are passed to apilogs via the same mechanism as the AWS CLI.

Conclusion

I hope apilogs can become part of your standard dev, test, or ops workflows. Check out apilogs on Github. Feedback and contributions are always welcome!