Injecting Failures and Latency using the AWS SDK for Java

Today we have another guest post from a member of the Amazon DynamoDB team, Pejus Das.

The Amazon DynamoDB service provides fast and predictable performance with seamless scalability. It also has a list of common errors that can occur during request processing. You probably have a set of test suites that you run before you release changes to your application. If you’re using DynamoDB in your application, your tests probably call it using an isolated test account, or one of the sets of mock DynamoDB facades out there. This link lists some sample open source libraries, Object-Relational Mappers, and Mock implementations. Or maybe you have a combination of both solutions, with mocking for unit tests and using a test account for integration tests. Either way, your test suite likely covers expected successful scenarios, and expected failure scenarios.

But then there are the other classes of failures that are harder to test for. Amazon DynamoDB is a remote dependency that you call across the network (or possibly even over the internet). A whole class of things can go wrong with this kind of an interaction, and when things do go wrong, your application will behave a lot better if you’ve tested those failure scenarios in advance.

There are many approaches to injecting unexpected failures in your application. For example, you can simulate what happens to your application when DynamoDB returns one of the documented errors returned by DynamoDB. You can also test the impact of high request latencies on your application. Such testing helps to build reliable and robust client applications that gracefully handle service errors and request delays. In this blog post, we describe another approach: how you can easily inject these kinds of failures into the client application using the AWS SDK for Java.

Request Handlers

The AWS SDK for Java allows you to register request handlers with the DynamoDB Java client. You can attach multiple handlers and they are executed in the order you added them to the client. The RequestHandler interface gives you three hooks into the request execution cycle: beforeRequest, afterRequest, and afterError.

Hook	Description
beforeRequest	Called just before executing the HTTP request against an AWS service like DynamoDB
afterRequest	Called just after the Response is received and processed by the Client
afterError	Called if there are any AmazonClientException errors while executing the HTTP request

The RequestHandler hooks give an easy way to inject failures and latencies in the client for testing.

Injecting Failures

The beforeRequest hook provides access to the Request object. You can inspect the Request and take some action based either on the Request or on some other condition. In the following example, we inspect a PutRequest and inject a ProvisionedThroughputExceededException on an average 50 percent of the time.

@Override
public void beforeRequest(Request<?> request) {
    // Things to do just before a request is executed 
    if (request.getOriginalRequest() instanceof PutItemRequest) {
        // Throw throuhgput exceeded exception for 50% of put requests 
        if (rnd.nextInt(2) == 0) {
           logger.info("Injecting ProvisionedThroughputExceededException");
           throw new ProvisionedThroughputExceededException("Injected Error");
        }
    }
    // Add latency to some Get requests 
    if (request.getOriginalRequest() instanceof GetItemRequest) {
        // Delay 50% of GetItem requests by 500 ms 
        if (rnd.nextInt(2) == 0) {
            // Delay on average 50% of the requests from client perspective 
            try {
                logger.info("Injecting 500 ms delay");
                Thread.sleep(500);
            } catch (InterruptedException ie) {
                logger.info(ie);
                throw new RuntimeException(ie);
            }
        }
    }
}

Injecting Latency

You could simply put a sleep in the beforeRequest hook to simulate latencies. If you want to inspect the Response object and inject latencies for specific traffic you would use the afterResponse hook. You can analyze the response data from DynamoDB and act accordingly. In the following example, we inspect for a GetItemRequest and when the item is an Airplane, we modify the item and additionally add a 500 ms delay.

@Override
public void afterResponse(Request<?> request, Object resultObject, TimingInfo timingInfo) {
    // The following is a hit and miss for multi-threaded
    // clients as the cache size is only 50 entries
    String awsRequestId = dynamoDBClient.getCachedResponseMetadata(
                          request.getOriginalRequest()).getRequestId();
    logger.info("AWS RequestID: " + awsRequestId);
    // Here you could inspect and alter the response object to
    // see how your application behaves for specific data
    if (request.getOriginalRequest() instanceof GetItemRequest) {
        GetItemResult result = (GetItemResult) resultObject;
        Map item = result.getItem();
        if (item.get("name").getS().equals("Airplane")) {
            // Alter the item
            item.put("name", new AttributeValue("newAirplane"));
            item.put("new attr", new AttributeValue("new attr"));
            // Add some delay
            try {
                Thread.sleep(500);
            } catch (InterruptedException ie) { 
                logger.info(ie);
                throw new RuntimeException(ie);
            }
        }
    }
}

The preceding code examples are listed on GitHub in the aws-labs repository here.

While this approach simulates increased latency and failures, it is only a simulation based on what you think will be happening during a failure. If you want to test your application for failures even more thoroughly, take a look at the Chaos Monkey and Simian Army applications written by Netflix. These inject actual failures into your system, revealing the interactions between more components in your system than just your application logic. We hope that adding fault injection testing to your application helps you be prepared for failure. Let us know in the comments!

AWS Developer Tools Blog

Injecting Failures and Latency using the AWS SDK for Java

Request Handlers

Injecting Failures

Injecting Latency

Reference Links

Resources

Follow