Concurrency in Version 3 of the AWS SDK for PHP

From executing API commands in the background to grouping commands and waiters into concurrent pools, version 3 of the AWS SDK for PHP lets you write asynchronous code that blocks only when you need it to. In this blog post, I’ll show you how to take advantage of some of the SDK’s concurrency abstractions, including promises, command pools, and waiters.

Promises

The AWS SDK for PHP makes extensive use of promises internally, relying on the Guzzle Promises library. Each operation defined on an API supported by the SDK has an asynchronous counterpart that can be invoked by tacking Async to the name of the operation. An asynchronous operation will immediately return a promise that will be fulfilled with the result of the operation or rejected with an error:

$s3Client = new AwsS3S3Client([ 
    ‘region’ => ‘us-west-2’,
    ‘version’ => ‘2006-03-01’,
]);

// This call will block until a response is obtained
$buckets = $s3Client->listBuckets();

// This call will not block
$promise = $s3Client->listBucketsAsync();

// You can provide callbacks to invoke when the request finishes
$promise->then(
    function (AwsResultInterface $buckets) {
        echo ‘My buckets are: ‘
             . implode(‘, ‘, $buckets->search(‘Buckets[].Name’));
    },
    function ($reason) {
        echo "The promise was rejected with {$reason}";
    }
);

The Guzzle promises returned by asynchronous operations have a wait method that you can call if you need to block until an operation is completed. In fact, synchronous operations like $s3Client->listBuckets() are calling asynchronous operations under the hood and then waiting on the result.

Where promises really shine, though, is in groupings. Let’s say you need to upload ten files to an Amazon S3 bucket. Rather than simply loop over the files and upload them one by one, you can create ten promises and then wait for those ten requests to be complete:

$filesToUpload = [
    ‘/path/to/file/1.ext’,
    ‘/path/to/file/2.ext’,
    …
    ‘/path/to/file/10.ext’,
];
$promises = [];
foreach ($filesToUpload as $path) {
    $promises []= $s3Client->putObjectAsync([
        ‘Bucket’ => $bucketName,
        ‘Key’ => basename($path),
        ‘SourceFile’ => $path,
    ]);
}
// Construct a promise that will be fulfilled when all
// of its constituent promises are fulfilled
$allPromise = GuzzleHttpPromiseall($promises);
$allPromise->wait();

Rather than taking ten times as long as uploading a single file, the asynchronous code above will perform the uploads concurrently. For more information about promises, see the AWS SDK for PHP User Guide.

Waiters

Some AWS operations are naturally asynchronous (for example, those in which a successful response means that a process has been started, but is not necessarily complete). Provisioning Amazon EC2 instances or S3 buckets are gppd examples. If you were starting a project that required three S3 buckets and an Amazon ElastiCache cluster, you might start out by provisioning those resources programmatically:

$sdk = new AwsSdk([‘region’ => ‘us-west-2’, ‘version’ => ‘latest’]);
$elasticacheClient = $sdk->get(‘elasticache’);
$s3Client = $sdk->get(‘s3’);
$promises = [];
for ($i = 0; $i < 3; $i++) {
    $promises []= $s3Client->createBucket([
        ‘Bucket’ => “my-bucket-$i”,
    ]);
}
$cacheClusterId = uniqid(‘cache’);
$promises []= $elasticacheClient->createCacheCluster([
    ‘CacheClusterId’ => $cacheClusterId,
]);
$metaPromise = GuzzleHttpPromiseall($promises);
$metaPromise->wait();

Waiting on the $metaPromise will block only until all of the requests sent by the createBucket and createCacheCluster operations have been completed.You would need to use a waiter to block until those resources are available. For example, you can wait on a single bucket with an S3Client’s waitUntil method:

$waiterPromises = [];
for ($i = 0; $i < 3; $i++) {
    // Create a waiter
    $waiter = $s3Client->getWaiter(‘BucketExists’, [
        ‘Bucket’ => “my-bucket-$i”,
    ]);
    // Initiate the waiter and retrieve a promise.
    $waiterPromises []= $waiter->promise();
}
$waiterPromises []= $elasticacheClient
    ->getWaiter(‘CacheClusterAvailable’, [
        ‘CacheClusterId’ => $cacheClusterId,
    ])
    ->promise();
// Composer a higher-level promise from the individual waiter promises
$metaWaiterPromise = GuzzleHttpPromiseall($waiterPromises);
// Block until all waiters have completed
$metaWaiterPromise->wait();

Command Pools

The SDK also allows you to use command pools to fine-tune the way in which a series of operations are performed concurrentl. Command pools are created with a client object and an iterable list of commands, which can be created by calling getCommand with an operation name on any SDK client:

// Create an S3Client
$s3Client = new AwsS3S3Client([
    ‘region’ => ‘us-west-2’,
    ‘version’ => ‘latest’,
]);

// Create a list of commands
$commands = [
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket1’]),
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket2’]),
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket3’]),
];

// Create a command pool
$pool = new AwsCommandPool($s3Client, $commands);

// Begin asynchronous execution of the commands
$promise = $pool->promise();

// Force the pool to complete synchronously
$promise->wait();

How is this different from gathering promises from individual operations, such as by calling $s3Client->listObjectsAsync(…)? One key difference is that no action is taken until you call $pool->promise(), whereas requests are dispatched immediately when you call $s3Client->listObjectsAsync(…). The CommandPool defers the initiation of calls until you explicitly tell it to do so. In addition, by default, a command pool will limit concurrency to 25 operations at a time by default, This simultaneous operation limit can be tuned according to your project’s needs. For a more complex example see CommandPool in the AWS SDK for PHP User Guide.

With promises, waiters, and command pools, version 3 of the AWS SDK for PHP makes it easy to write asynchronous or concurrent code! We welcome your feedback.