AWS Machine Learning Blog
Create a Word-Pronunciation sequence-to-sequence model using Amazon SageMaker
Amazon SageMaker seq2seq offers you a very simple way to make use of the state-of-the-art encoder-decoder architecture (including the attention mechanism) for your sequence to sequence tasks. You just need to prepare your sequence data in recordio-protobuf format and your vocabulary mapping files in JSON format. Then you need to upload them to Amazon Simple Storage Service (Amazon S3). The Amazon SageMaker built-in algorithm handles building and training the deep learning (DL) architecture for you.
In this blog post, we walk you through our sample Amazon SageMaker Word-Pronunciation notebook to get you familiarized with SageMaker seq2seq. In the English language, there is an implicit but complicated rule to decode the pronunciation of a word from its spelling. We are going to model the rule using sequences of letters of the alphabet as source input and their corresponding sequences of phonemes as target output. This kind of model is referred as sequence to sequence.
The application of the seq2seq model is not limited to the relationship between word and pronunciation. Other use cases would be English sentences to German sentences, texts to summaries, texts to titles, questions to answers, and so on. This blog post will show you how to deploy your own sequence data and develop your own custom seq2seq model on Amazon SageMaker.
Amazon SageMaker seq2seq is based on the Sockeye package, which is a framework for seq2seq modeling with Apache MXNet. For more information see this AWS Blog. However, it uses a different input data format and renames some hyperparameters to work effectively in Amazon SageMaker.
Data: Carnegie Mellon University Pronunciation Dictionary (cmudict-0.7b)
In this blog post, we use the Carnegie Mellon University Pronunciation Dictionary (cmudict-0.7b) data. A phonetic dictionary, which is a mapping of vocabulary words to sequences of phonemes, is important in a Text-to-Speech system. The original raw data will be found through the following link:
Dataset: http://www.speech.cs.cmu.edu/cgi-bin/cmudict
cmudict-0.7b is a pronunciation dictionary for North American English, which contains more than 134,000 word-pronunciation pairs. One of the advantages of using this dataset for our sequence-to-sequence task is that we can easily validate the results quantitatively and qualitatively.
A pair of word-pronunciation sequences looks like the following example.
In this example, the source word “Acknowledgement” is made up of 15 letters of the alphabet while the corresponding target pronunciation consists of 11 phonemes. Some words in the dataset have much longer sequences of letters and phonemes than “Acknowledgement” while some words are much shorter than it. The maximum lengths of words and phonemes in the dataset are 34 and 32, respectively.
In this blog post we show you how to train a seq2seq model using the cmudict-0.7b dataset so that the model can predict a sequence of phonemes out of a sequence of input letters.
The sample Amazon SageMaker Word Pronunciation Example can be found here:
Upload SageMaker-Seq2Seq-word-pronunciation.ipynb to your notebook instance. We also use create_vocab_proto.py and record_pb2.py because SageMaker-Seq2Seq-word-pronunciation.ipynb calls these two Python scripts.
Preprocess the dataset
The seq2seq model takes two types of sequence data: source sequences and their paired target sequences. In our example, the source and target are the word and its corresponding pronunciation. Each word is a sequence of letters of the alphabet and each pronunciation is a sequence of phonemes. The model also requires string-to-integer mappings that tokenize human readable sequences to machine readable sequences.
Amazon SageMaker seq2seq expects the following four input files: vocab.src.json, vocab.trg.json, train.rec, and val.rec. vocab.src.json, and vocab.trg.json are the string-to-integer vocabulary mappings in the json format. train.rec and val.rec are recordio-protobuf files that contain tokenized source-target pairs for training and validation data, respectively. We are going to generate those files below.
Vocabulary files
After downloading the dataset, tokenize the letters of the alphabet and the phonemes into numbers, as shown in the following example.
The letter A is mapped in 14, C is mapped in 34, K is mapped in 61, N is mapped in 64, and so on. This string-to-integer mapping becomes the vocabulary used in the JSON files. In this example, we create a joint string-to-integer mapping for both source and target sequences (for example, K = 61 and N = 64), which is used to generate both vocab.src.json, and vocab.trg.json. The following example demonstrates this:
In this example, ninety three letter-phoneme sets are mapped. You may notice that the first four indices are used for special characters (such as <pad> : pad, <unk> : unknown character, <s> : beginning of the sequence, and </s> end of the sequence). Amazon SageMaker seq2seq expects this input format. Thus, our own vocabulary mapping must start from the index number of 4.
Here is how we generate source and target JSON vocabulary files.
Now we have vocab.src.json, and vocab.trg.json.
Recordio-protobuf files
We are going to generate the recordio-protobuf files of training and validation data. The recordio-protobuf is the standard data I/O format expected by Amazon SageMaker algorithms. In the notebook, we have a helper function called “write_to_file” for you.
The write_to_file takes stacks of source sequences (e.g., trainX <numpy array>) and their pair-wise target sequences (e.g., trainY <numpy array>) as inputs and then yields one output protobuf file (e.g., train.rec). Stacks of sequences look like the examples that follow.
The recordio-protobuf compresses the data into a smaller size. This helps cut down time taken for data transfer. Our write_to_file helper function borrowed several sub functions from create_vocab_proto.py and record_pb2.py that are found in the following link:
After we have vocab.src.json, vocab.trg.json, train.rec, and val.rec, we are ready to upload them to Amazon S3 buckets.
Upload 4 files to Amazon S3
The S3 bucket and folder prefix are specified. The Region of the S3 bucket should be the same Region as the notebook and training and inference hosting instances. Name the folder prefix “seq2seq/word-pronunciation”.
Upload the train.rec to train folder, val.rec to validation folder, and the two vocabulary files to vocab folder.
def upload_to_s3(bucket, prefix, channel, file):
s3 = boto3.resource('s3')
data = open(file, "rb")
key = prefix + "/" + channel + '/' + file
s3.Bucket(bucket).put_object(Key=key, Body=data)
upload_to_s3(bucket, prefix, 'train', 'train.rec')
#/<your s3 bucket>/seq2seq/word-pronunciation/train/train.rec
upload_to_s3(bucket, prefix, 'validation', 'val.rec')
#/<your s3 bucket>/seq2seq/word-pronunciation/validation/val.rec
upload_to_s3(bucket, prefix, 'vocab', 'vocab.src.json')
#/<your s3 bucket>/seq2seq/word-pronunciation/vocab/vocab.src.json
upload_to_s3(bucket, prefix, 'vocab', 'vocab.trg.json')
#/<your s3 bucket>/seq2seq/word-pronunciation/vocab/vocab.trg.jsonThe following screenshot shows the training data file (train.rec) in Amazon S3.
The following screenshot shows the validation data file (val.rec) in Amazon S3.
The following screenshot shows the vocabulary data files (vocab.src.json and vocab.trg.json) in Amazon S3.
Training the Word Pronunciation model
Note: The training takes approximately two hours on an ml.p2.xlarge training instance.
First, we obtain a built-in training image from a Docker container. This training image contains the encoder-decoder model architecture, which is used for our sequence-to-sequence task. This is where the magic happens.
Then, we create a dictionary that passes all training parameters, such as file locations, instructions for training instance, and other hyper parameters, to the training job.
Inside the seq2seq model, there are two blocks: encoder and decoder. Each block consists of embedding and (multiple) recurrent neural network (RNN) layers. You can find the corresponding instructions for those layers in the “HyperParameters” key covered earlier (i.e., “rnn_num_hidden”: “512”, “num_layers_encoder”: “1”, “num_layers_decoder”: “1”, “num_embed_source”: “512”, “num_embed_target”: “512”). In addition, the attention mechanisms described in the paper by Luong et al. as concat are already implemented in the SageMaker seq2seq model as a default. The default value of the parameter “rnn_attention_type” is mlp, which is referred as concat in Luong et al. Likewise, the value bilinear is referred as general in the paper.
Look at “InputDataConfig”. That is where all locations of input files are instructed.
After the training is initiated, monitor the training status by executing the following command.
If it returns InProgress, the training is still ongoing. It takes approximately two hours on an ml.p2.xlarge training instance with the parameters given earlier. After it returns Completed, the training is successfully over. You will find an output model artifact (model.tar.gz) in your Amazon S3 bucket.
Here is the inside of the model.tar.gz.
symbol.json is a model structure file, whereas params.best is the model weights. decode.source and decode.target are text files in which tokenized source and target sequences are described line by line. They are constructed from the validation dataset. You can also choose the sample size of decode.source and decode.target by using the “bleu_sample_size” parameter. A series of decode.output.XXXXX keeps track of the prediction outputs at every checkpoint. The metrics files also keep track the change in training and validation accuracies.
Hosting inference
We now use the trained model to perform inference. There are three steps. First, we create an Amazon SageMaker model from the training output.
Then, we create the endpoint configuration. The endpoint configuration also contains information about the type and number of EC2 inference instances to use when hosting a model. Although we use the free-tier ml.m4.xlarge CPU instance for hosting in this example, we recommend the ml.p2xlarge GPU instance, which is more stable and can process requests much faster.
Finally, we create the endpoint that can be validated and incorporated into production applications. It takes about 10-15 minutes initially to set up the endpoint. After the endpoint is set up, you can use it for subsequent requests without any additional delay.
Here is how to perform inference. We know that the words “tapeworm” and “tapdance” are not listed in the original cmudict-0.7b dataset. And needless to say, the words “supercalifragilistic” or “expialidocious” are not listed either.
Here is the inference result.
Conclusion
Amazon SageMaker seq2seq is a supervised learning algorithm in which the input is a sequence of tokens and the output generated is another sequence of tokens. We walked through our end-to-end sample notebook. We trained a word-pronunciation model using the cmudict-0.7b dataset and hosted the inference. You may want to try out a different set of hyperparameters to examine the changes in the prediction accuracy. Or check out Amazon Polly, an API-driven service that enables you to quickly integrate speech into your applications, if you are looking for a plug-and-play solution. Amazon Polly supports lexicons that allow you to easily customize the pronunciation of words.
Now you can develop your custom seq2seq model by bringing in your own paired sequence data (for example, text summarization, Q&A modeling, title generation from text, machine translation, and so on). You are now able to generate train.rec, val.rec, vocab.src.json, and vocab.trg.json, upload them to Amazon S3, and run the training. The Amazon SageMaker notebook has already provided a sample seq2seq that can help you build an English-German machine translation model based on language data provided by the Machine Translation Group at UEDIN . We encourage you to take a look at the sample notebook as well. Or try Amazon Translate, a neural machine translation service that is currently in Preview.
Additional Reading
In this post, we provide an overview of NMT, and then show how to use Sockeye to train a minimal NMT model with attention.
About the Authors
Tatsuya Arai Ph.D. is a biomedical engineer turned deep learning data scientist in Amazon Machine Learning Solutions Lab team. He believes in the true democratization of AI and the power of AI shouldn’t be exclusive to computer scientists or mathematicians.
Orchid Majumder is a Software Engineer within the SageMaker Algorithms Team. Previously he worked as a Software Engineer in Amazon India Seller Experience Team. In his leisure time, he like reading fictions or watching cricket.
Saurabh Gupta is an Applied Scientist with AWS Deep Learning. He did his MS in AI and Machine Learning from UC San Diego. He is currently working on building Natural Language Processing algorithms for Amazon SageMaker.
David Arpin is AWS’s AI Platforms Selection Leader and has a background in managing Data Science teams and Product Management.
Sunil Mallya is a Senior Solutions Architect in the AWS Deep Learning team. He helps our customers build machine learning and deep learning solutions to advance their businesses. In his spare time, he enjoys cooking, sailing and building self driving RC autonomous cars.