Powering the Next Generation of AI Workloads on Amazon EKS with Anyscale

Ray is an open-source framework that manages, executes, and optimizes compute needs for AI workloads. It is designed to make it easy to write parallel and distributed Python applications by providing a simple and intuitive API for distributed computing. Ray unifies infrastructure by leveraging any compute instance and accelerator on AWS via a single, flexible framework—enabling any AI workload from data processing to model training to model serving and beyond.

Ray, launched as a research project in 2016, is quickly becoming a standard for orchestrating AI workloads, especially with Kubernetes. On AWS, Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that makes it easier for you to run Kubernetes on AWS without installing and operating your own Kubernetes control plane or worker nodes. Amazon EKS is deeply integrated with key AWS services to provide compute, scalability, and security for applications. EKS is well-suited for running AI and machine learning (ML) workloads, providing a scalable and reliable infrastructure for deploying and managing AI/ML models and applications.

Together EKS and Ray provide key components that let customers quickly build and scale AI applications in the cloud. This week, at Ray Summit 2024, Anyscale released a slew of new features including RayTurbo, a new optimized version of Ray with 4.5x faster data processing and optimizations which let customers run up to 50% fewer nodes for online model serving.

What’s even more exciting is that Anyscale has integrated RayTurbo natively with Kubernetes. Customers will be able to connect their EKS clusters to Anyscale’s SaaS management platform to install and manage RayTurbo across their Amazon EKS clusters. This will allow you to take advantage of reliable and scalable AWS compute infrastructure for machine learning, including AWS Trainium and AWS Inferentia chips in fully managed Ray and EKS clusters.

The powerful combination of EKS and RayTurbo gives customers next generation performance, reliability, and ease of use for training and running AI applications.

Next Steps

Today, RayTurbo on EKS is available from Anyscale to select customers. Please reach out to your AWS account team or book a demo directly with Anyscale. Over the next few months Anyscale and AWS will work closely to bring this experience to more customers to enable push button AI infrastructure on Amazon EKS.