Posted On: Nov 24, 2020
Amazon Managed Workflows is a new managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows”.
Organizations want to leverage massive amounts of data for analytics and insights. To effectively use this data, customers often need to first build a workflow that defines the series of sequential tasks that prepare and process the data. Many customers use Apache Airflow to take advantage of its open source community and large library of pre-built integrations to third-party data processing tools such as Apache Spark and Hadoop. With Apache Airflow, workflows are created as Python scripts that are familiar to data engineers and data scientists. However, data engineers need to install, maintain, scale, and secure Airflow themselves, which adds cost and operational complexity.
Now, data engineers and data scientists can use the same open source Airflow platform and Python language to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Managed Workflows automatically scales its workflow execution capacity to meet your needs, and is integrated with AWS security services to enable fast and secure access to data.
There is no up-front cost commitment or minimum fee for using Managed Workflows , and you pay only for what you use.
You can get started with Managed Workflows using the AWS Management Console, Command Line Interface (CLI), AWS CloudFormation or SDK. You can create an account and begin deploying Directed Accyclic Graphs (DAGs) to your Airflow environment immediately without reliance on development resources or provisioning infrastructure.
Managed Workflows is generally available today in the following regions: US East (Ohio and N. Virginia), US West (Oregon), EU (Stockholm, Ireland, and Frankfurt), and Asia Pacific (Tokyo, Singapore, and Sydney), with more regions to follow.
To learn more about Managed Workflows:
- Read Jeff Barr's blog post
- Visit the Amazon Managed Workflows product page
- Read the Amazon Managed Workflows User Guide