Tag: Data on EKS

Many organizations are building artificial intelligence (AI) applications using Large Language Models (LLMs) to deliver new experiences to their customers, from content creation to customer service and data analysis. However, the substantial size and intensive computational requirements of these models may have challenges in configuring, deploying, and scaling them effectively on graphic processing units (GPUs). […]

In Part 1, we explored Moka’s high-level design and logging infrastructure, showcasing how AWS for Fluent Bit, Amazon S3, and a robust logging framework make sure of operational visibility and facilitate issue resolution. For more details, read part 1 here. Introduction As we transition to the second part of our series, our focus shifts to […]

Introduction Quora is a leading Q&A platform with a mission to share and grow the world’s knowledge, serving hundreds of millions of users worldwide every month. Quora uses machine learning (ML) to generate a custom feed of questions, answers, and content recommendations based on each user’s activity, interests, and preferences. ML drives targeted advertising on […]

Introduction Apache Kafka, a distributed streaming platform, has become a popular choice for building real-time data pipelines, streaming applications, and event-driven architectures. It is horizontally scalable, fault-tolerant, and performant. However, managing and scaling Kafka clusters can be challenging and often time-consuming. This is where Kubernetes, an open-source platform for automating deployment, scaling, and management of […]

Introduction In recent years, there’s been a remarkable surge in the adoption of Kubernetes for data analytics and machine learning (ML) workloads in the tech industry. This increase is underpinned by a growing recognition that Kubernetes offers a reliable and scalable infrastructure to handle these demanding computational workloads. Furthermore, a recent wave of Generative AI […]

This post was co-written with Derek Brown, Infrastructure Engineer, Lacework Introduction Lacework is a security platform that automatically monitors and detects misconfigurations and security vulnerabilities across our customers’ cloud environments. Lacework takes a data-driven approach to alerting and detection, consuming trillions of events from a huge list of sources including eBPF, Kubernetes Audit Logs, cloud […]

Introduction We are thrilled to introduce Data on EKS (DoEKS), a new open-source project aimed at streamlining and accelerating the process of building, deploying, and scaling data workloads on Amazon Elastic Kubernetes Service (Amazon EKS). With DoEKS, customers get access to a comprehensive range of resources including Infrastructure as Code (IaC) templates, performance benchmark reports, […]

Introduction Kubernetes has gained widespread adoption in the field of data processing because of its ability to package and deploy applications as containers with all required dependencies, as well as its support for running data frameworks. This makes it easy for developers to run their Data Analytics/Machine Learning (ML) applications within a Kubernetes cluster and […]