Build a real-time leaderboard with Amazon Aurora Serverless and Amazon ElastiCache

Introduction

Background and setup

Provision a database

Plan your data model

Launch a cluster

Design your data model

Add authentication to your application

Deploy your application

Test your application

Clean up and next steps

Design your Redis data model

In this module, you design the data model for use in your application.

When using an in-memory data store like Redis, your main goal is speed and throughput. You want to serve a high number of requests as quickly as possible. To accomplish this, you organize your data in use-case specific ways. Rather than designing your data to be queried flexibly, like in a relational database, you arrange your data so that it handles your needs as quickly as possible. When designing your data model with Redis, you need to think about your needs and access patterns first.

For your leaderboard application, you want to quickly access the highest scores around the world. You have three main access patterns for your data:

  1. Fetch the highest scores for all time;
  2. Fetch the highest scores in the last month;
  3. Fetch the highest scores for a given day.

To handle these access patterns, you can use Redis Sorted Sets. Sorted Sets are a data structure that enables fast, ordered lookups of values. You can save the highest score for a particular user in a Sorted Set, then quickly retrieve the top 5 scores in the Sorted Set.

In your application, you create a Sorted Set for each potential access pattern you need -- overall, by week, and by day. As a new high score comes in, you load it into the relevant Sorted Sets. As users request the top scores, you can query for the top scores from the proper Sorted Set.

In the following steps, you load your sample data into your ElastiCache Redis instance and query your data for the top scores.

Time to Complete Module: 20 Minutes

  • Step 1. Load sample data into your Redis instance

    Step 1. Load sample data into your Redis instance

    First, you load your sample data into a Redis instance. You use the same 300 games that you loaded into your Amazon Aurora Serverless instance in a previous module.

    For each game you load, you want to put it into three different Sorted Sets:

    1. The overall leaderboard;
    2. The monthly leaderboard for the game; and
    3. The daily leaderboard for the game.

    These separate objects in Redis allow for the most efficient queries at read time.

    In the scripts/ directory, there is a file called loadRedis.js. The contents of that are as follows:

    const redis = require('redis')
const fs = require('fs');
const path = require('path');

const raw = fs.readFileSync(path.resolve( __dirname, 'games.json'));
const games = JSON.parse(raw)

const client = redis.createClient({
  url: `redis://${process.env.REDIS_ENDPOINT}`
})
client.on('error', function(err) {
  console.log('Received Redis error:', err)
})

games.forEach((game) => {
  const gametime = new Date(game.gamedate)
  const key = `${game.username}|${game.gamedate}|${game.level}`
  client.multi()
    .zadd('Overall Leaderboard', game.score, key)
    .zadd(`Monthly Leaderboard|${gametime.getUTCMonth()}-${gametime.getUTCFullYear()}`, game.score, key)
    .zadd(`Daily Leaderboard|${gametime.getUTCDay()}-${gametime.getUTCMonth()}-${gametime.getUTCFullYear()}`, game.score, key)
    .exec((err) => {
      if (err) {
        console.log('Error: ', err);
      }
    })
})

console.log('Loaded data!')

client.quit()

    Like the pingRedis.js script, you are creating a Redis client. Then, you read all of the game data into memory. For each game element, you make three writes to Redis -- one for the overall leaderboard, one for the weekly leaderboard, and one for the daily leaderboard. Each write is using the ZADD command in Redis to add an element to a Sorted Set.

    Notice the structure of both the overall Sorted Set key name, as well as the name of the element within the Sorted Set.

    For the Sorted Set key names, the overall leaderboard is simply Overall Leaderboard. For the monthly and daily leaderboard, the date information is included in it. Thus, the monthly leaderboard key name for November 2019 is Monthly Leaderboard|11-2019, and the daily leaderboard key name for November 8, 2019 is Daily Leaderboard|8-10-2019. This naming scheme allows you to find the proper leaderboard key at read time.

    Now, look at the key name when an element is inserted into the table. It is a combination of username, the timestamp, and the level. Encoding all of this information into the key name allows your application to show additional information about the high score when it fetches the overall high scores.

    Execute the loadRedis.js script with the following command:

    node scripts/loadRedis.js

    You should see the following output in your terminal:

    Loaded data!

    In the next step, you learn how to read the top scores from a key.

  • Step 2. Read the top scores from a Sorted Set

    Step 2. Read the top scores from a Sorted Set

    Now that you have loaded some sample data into your table, you can read some data from your table.

    To query the top elements in a Sorted Set, you can use the ZREVRANGE command in Redis. You specify the name of the Sorted Set you want to query, the positions in the Sorted Set to return, and whether you want the scores returned as well.

    In the scripts/ directory, there is a file called getTopOverallScores.js. The contents of that file are as follows:

    const redis = require('redis')
const _ = require('lodash')
const client = redis.createClient({
  url: `redis://${process.env.REDIS_ENDPOINT}`
})

client.on('error', function(err) {
  console.log('Received Redis error:', err)
})

const parseKey = (key) => {
  const parts = key.split('|')
  return {
    username: parts[0],
    gamedate: parts[1],
    level: parts[2]
  }
}

const parseZRevRangeResponse = (resp) => {
  const result = _.chunk(resp, 2).map(([key, score]) => {
    const obj = parseKey(key)
    return {
      ...obj,
      score
    }
  })
  return result
}

client.zrevrange('Overall Leaderboard', '0', '4', 'WITHSCORES', (err, resp) => {
  if (err) {
    console.log('Error reading leaderboard: ', err);
  } else {
    const scores = parseZRevRangeResponse(resp)
    console.log('Top overall scores:')
    console.log(scores)
  }
})

client.quit()

    This script runs the ZREVRANGE command to fetch the top five scores from the overall leaderboard.

    Notice the code in the callback to the ZREVRANGE command. The results from Redis are returned in an array. The array first has a key element, then a score element, then the next key element, etc. This isn’t the most helpful way to return this to the frontend, so the parseZRevRangeResponse helps to parse that into something more meaningful.

    Execute the script with the following command:

    node scripts/getTopOverallScores.js

    You should see the following output in your terminal:

    Top overall scores:
[ { username: 'debbieschneider',
    gamedate: '2019-11-09T18:41:27',
    level: '28',
    score: '9895' },
  { username: 'alicia39',
    gamedate: '2019-11-09T10:39:59',
    level: '47',
    score: '9824' },
  { username: 'rosecolleen',
    gamedate: '2019-11-10T07:09:51',
    level: '58',
    score: '9765' },
  { username: 'allisonsandra',
    gamedate: '2019-11-07T22:43:32',
    level: '62',
    score: '9760' },
  { username: 'kathrynmorris',
    gamedate: '2019-11-05T04:31:37',
    level: '85',
    score: '9722' } ]

    The script should print out the top five scores as objects with username, game date, level, and score properties.

Conclusion

In this module, you learned about modeling your data in ElastiCache for fast lookups. Then, you loaded your sample data into multiple Sorted Sets to satisfy your use cases. Finally, you saw how to read the top items from a Sorted Set.

In the next module, you configure Amazon Cognito to add authentication to your application.

Next: Add authentication to your application