AWS Machine Learning Blog
Automatically extract text and structured data from documents with Amazon Textract
September 2022: Post was reviewed for accuracy.
December 2021: This post has been updated with the latest use cases and capabilities for Amazon Textract.
September 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.
Documents are a primary tool for record keeping, communication, collaboration, and transactions across many industries, including financial, medical, legal, and real estate. The millions of mortgage applications and hundreds of millions of W2 tax forms processed each year are just a few examples of such documents. A lot of information is locked in unstructured documents. It usually requires time-consuming and complex processes to enable search and discovery, business process automation, and compliance control for these documents.
In this post, we show how you can take advantage of Amazon Textract to automatically extract text and data from scanned documents without any machine learning (ML) experience. While AWS takes care of building, training, and deploying advanced ML models in a highly available and scalable environment, you take advantage of these models with simple-to-use API actions. We cover the following use cases in this post:
- Text detection from documents
- Form and table extraction and processing
- Extract information from identity documents
- Extract information from invoices and receipts
- Multi-column detection and reading order
- Natural language processing and document classification
- Natural language processing for medical documents
- Document translation
- Search and discovery
- Compliance control with document redaction
- PDF and multi-page TIFF document processing
Amazon Textract overview
Before we get started with the use cases, let’s review and introduce some of the core features. Amazon Textract goes beyond simple optical character recognition (OCR) to also identify the contents of fields in forms, information stored in tables, handwritten text, and check boxes. This allows you to use Amazon Textract to instantly read almost any type of document and accurately extract text and data without the need for any manual effort or custom code.
The following images show an example document using Amazon Textract on the AWS Management Console on the Forms output tab.
To quickly download a .zip file containing the output, choose Download results. You can choose various formats, including raw JSON, text, and CSV files for forms and tables.
In addition to the detected content, Amazon Textract provides additional information like confidence scores and bounded boxes for detected elements. It gives you control of how you consume extracted content and integrate it into various business applications.
Amazon Textract provides both synchronous and asynchronous API actions to extract document text and analyze the document text data. Synchronous APIs can be used for single-page documents and low-latency use cases such as mobile capture. Asynchronous APIs can be used for multipage documents such as PDF or TIFF documents with thousands of pages. For more information, see the Amazon Textract API Reference.
Use cases overview
You can easily take advantage of Amazon Textract API operations using the AWS SDK to build power-smart applications. We also use Amazon Textract Helper, Amazon Textract Caller, Amazon Textract PrettyPrinter, and Amazon Textract Response Parser for some of the following use cases. These packages are published to PyPI to speed up development and integration even further.
Text detection from documents
We start with a simple example of how to detect text from a document. We use the following image as an input document to Amazon Textract. The sample image isn’t good quality, but Amazon Textract can still detect the text with accuracy.
The easiest way to extract information from this document programmatically is through installing Amazon Textract Helper:
Then we call Amazon Textract to extract information from the document and display the results by running the command line tool:
The following screenshot shows our output.
The command line tool uses the Amazon Textract Caller, Amazon Textract PrettyPrint, and Amazon Textract Overlayer packages to generate the results.
The original Amazon Textract response is in JSON format and has the following format:
By using Amazon Textract Response Parser, it’s easier to de-serialize the JSON response and use in your program, the same way Amazon Textract Helper and Amazon Textract PrettyPrinter use it. The GitHub repository shows some examples.
Form and table extraction and processing
Amazon Textract can provide the inputs required to automatically process forms and tables without human intervention. For example, a bank could write code to read PDFs of loan applications. The information contained in the document could be used to initiate all the necessary background and credit checks to approve the loan so that customers can get instant results for their application rather than having to wait several days for manual review and validation.
The following image is an employment application with form fields, check boxes, and a table.
The following code example extracts forms from the employment application and processes different fields:
The preceding commands produce the following output to visualize the structure of the information.
The key-value pairs from the FORMS
output are rendered as a table with Key
and Value
headlines to allow for easier processing.
For example, changing the output format by including —pretty-print-table-format=csv parameter
outputs the data in CSV format (check amazon-textract —help
for a list of other formats):
The following screenshot shows the output.
Amazon Textract can detect tables and their content. A company can extract all the amounts from an expense report (as in the following screenshot) and apply rules, such as any expense more than $1,000 needs extra review.
The following code uses the CSV output from the command line tool and the sample expense report to print the content of each cell, along with a warning message if any expense is more than $1,000:
Save this code as test-csv.py
or copy it from Amazon Simple Storage Service (Amazon S3) at s3://amazon-textract-public-content/blogs/test-csv.py
. Then use the following command:
We receive the following output.
To recap, we started with a document image, called Amazon Textract to identify and receive the table structure and information, applied business logic on the data, and triggered a business process based on the information.
Extract information from invoices and receipts
Invoices and receipts are difficult to process at scale because they follow no set design rules, yet any individual customer encounters thousands of distinct types of these documents. The Amazon Textract AnalyzeExpense action identifies standard fields and line-item details for these document types.
The standard fields supported include “Vendor Name”, “Total”, “Receiver Address”, “Invoice/Receipt Date”, “Invoice/Receipt ID”, “Payment Terms”, “Subtotal”, “Due Date”, “Tax”, “Invoice Tax Payer ID”, “Item Name”, “Item Price”, “Item Quantity” plus line-item details. For a complete list check the Analyzing Invoices and Receipts documentation.
The AWS Management Console offers options to test the AnalyzeExpense action through the “Select Document” options “Receipt” (image below) or “Invoice” or by “Choose File” option. The latter allows uploading of a document and subsequent selection of “Analyze Expense” in the output tab on the right side. Through “Download results” a zip file including the line-item fields and summary fields can be received.
You can call the AnalyzeExpense
API using the AWS Command Line Interface (AWS CLI), as shown in the following code. Make sure you have AWS CLI version >= 2.2.23 installed (check with aws --version
).
The output is the Textract JSON response.
We also created a Amazon Textract response parser library to parse the JSON returned by the AnalyzeExpense API. The library parses JSON and provides programming language-specific constructs to work with different parts of the document.
First install the dependencies.
This Python code takes the JSON response and prints out summary and line items in a table structure:
Output from code
More details and examples to the AnalyzeExpense action can be found in the blog post Announcing specialized support for extracting data from invoices and receipts using Amazon Textract.
Extract information from identity documents
Analyze ID helps you automatically extract information from identification documents such as driver’s licenses and passports. Using the following sample image we can use the amazon-textract-caller
and the amazon-textract-response-parser
to quickly extract the information from the document.
First install the dependencies.
tabulate
is only used for visualization purposes in this example and is not necessary for automation.
This script calls the Analyze ID API and prints out the values in a tabular format.
The output in this case is just the key and value pairs. Analyze ID also returns the confidence score and normalized values when available.
Multi-column detection and reading order
Traditional OCR solutions read left to right and don’t detect multiple columns, so they may generate incorrect reading order for multi-column documents. In addition to detecting text, Amazon Textract provides additional geometry information that you can use to detect multiple columns and print the text in reading order.
The following image is a two-column document. Similar to the earlier example, the image isn’t good quality, but Amazon Textract still performs well.
The following example code processes the document with Amazon Textract and takes advantage of geometry information to print the text in reading order:
The following image shows the output of the detected text in the correct reading order.
Natural language processing and document classification
Customer emails, support tickets, product reviews, social media, even advertising copy all represent insights into customer sentiment that can be put to work for your business. A lot of such content contains images or scanned versions of documents. After text is extracted from these documents, you can use Amazon Comprehend to detect sentiment, entities, key phrases, syntax, and topics. You can also train Amazon Comprehend to detect custom entities based on your business domain. You can then use these insights to classify documents, automate business process workflows, and ensure compliance.
The following example code processes the first image sample we used earlier with Amazon Textract to extract text and then uses Amazon Comprehend to detect sentiment and entities:
The following image shows the output text along with the text analysis from Amazon Comprehend. It found the sentiment to be neutral and detected “Amazon” as an organization, “Seattle, WA” as a location, and “July 5th, 1994” as a date, along with other entities.
Natural language processing for medical documents
An important way to improve patient care and accelerate clinical research is by understanding and analyzing the insights and relationships that are “trapped” in free-form medical text. These can include hospital admission notes and patient medical history.
In this example, we use the following document to extract text using Amazon Textract. You then use Amazon Comprehend Medical to extract medical entities, such as medical condition, medication, dosage, strength, and protected health information (PHI).
The following example code detects different medical entities:
The following image and text block shows the output of the detected text with information categorized by type. It detected “40yo” as the age with category Protected Health Information
. It also detected different medical conditions, including sleeping trouble, rash, inferior turbinates, and erythematous eruption. It recognized different medications and anatomy information.
Document translation
Many organizations localize content for international users, such as websites and applications. They must translate large volumes of documents efficiently. You can use Amazon Textract with Amazon Translate to extract text and data and then translate them into other languages.
The following code example shows translating the text in the first image to German:
The following image shows the output of the detected text, translated to German line by line.
Search and discovery
Extracting structured data from documents and creating a smart index using Amazon OpenSearch Service allows you to search through millions of documents quickly. For example, a mortgage company could use Amazon Textract to process millions of scanned loan applications in a matter of hours and have the extracted data indexed in Amazon ES. This would allow them to create search experiences like searching for loan applications where the applicant name is John Doe, or searching for contracts where the interest rate is 2%.
The following code example extracts text from the first image, stores it in Amazon ES, and searches it using Kibana:
The following image shows the output of extracted text in Kibana search results.
You can also build a custom UI experience by taking advantage of the Amazon ES APIs. Later in the post, you learn how to extract forms and tables and then index that structured data similarly to enable smart search.
Compliance control with document redaction
Because Amazon Textract identifies data types and form labels automatically, AWS helps secure infrastructure so that you can maintain compliance with information controls. For example, an insurer could use Amazon Textract to feed a workflow that automatically redacts personally identifiable information (PII) for review before archiving claim forms. Amazon Textract recognizes the important fields that require protection.
The following code example extracts all the form fields in the employment application used earlier and redacts all the address fields:
The following output is the redacted version of the employment application.
PDF or multi-page TIFF document processing (asynchronous API operations)
For the earlier examples, you used images with the synchronous API operations. Now, see how we process PDF files using the asynchronous API operations. Single page or multi-page TIFF documents are also supported in the asynchronous API operations.
With the amazon-textract
command line tool, you can pass in a PDF (the location for the PDF has to be on Amazon S3) and the underlying implementation calls the asynchronous API for StartDocumentTextDetection or StartDocumentAnalysis to start an Amazon Textract job:
The following screenshot shows our output.
When you use the asynchronous API from a Python program or the Python Interpreter, it looks like the following code:
We get the following output.
First, StartDocumentTextDetection or StartDocumentAnalysis is called to start an Amazon Textract job. Amazon Textract publishes the results of the Amazon Textract request, including completion status, to Amazon Simple Notification Service (Amazon SNS). You can then use GetDocumentTextDetection or GetDocumentAnalysis to get the results from Amazon Textract.
Conclusion
In this post, we showed you how to use Amazon Textract to automatically extract text and data from scanned documents without any ML experience. We covered use cases in fields such as finance, healthcare, and HR, but there are many other opportunities in which the ability to unlock text and data from unstructured documents could be useful.
You can start using Amazon Textract in the Regions US East (Ohio), US East (Northern Virginia), US West (N. California), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Canada (Central), EU (Frankfurt), EU (Ireland), EU (London), EU (Paris), AWS GovCloud (US-East), and AWS GovCloud (US-West).
To learn more about Amazon Textract, read about processing single-page and multipage documents, working with block objects, and code samples.
About the Authors
Kashif Imran is a Solutions Architect at Amazon Web Services. He works with some of the largest strategic AWS customers to provide technical guidance and design advice. His expertise spans application architecture, serverless, containers, NoSQL and machine learning.
Martin Schade is a Senior ML Product SA with the Amazon Textract team. He has 20+ years of experience with internet-related technologies, engineering and architecting solutions and joined AWS in 2014, first guiding some of the largest AWS customers on most efficient and scalable use of AWS services and later focused on AI/ML with a focus on computer vision and at the moment is obsessed with extracting information from documents.