Kubernetes, Alluxio and the disaggregated analytics stack
TL;DR: First the news - Alluxio support for K8s Helm charts now available! K8s is a certified environment for Alluxio. Now the take away- Alluxio brings back data locality for the disaggregated analytics stack in K8s. How? Read on.
There’s no arguing the rise of containers in real-world deployments over the past few years. Containers simplify running applications in any environment and Kubernetes further transforms the way software and applications are deployed and scaled agnostic of environments. In fact, Kubernetes is increasingly seen as a key technology that enables not only easy resource orchestration in the data center but also in hybrid and multi-cloud environments. While containers and Kubernetes works exceptionally well for stateless applications like web servers and even completely self-contained databases like mongoDB, Couchbase and others, the stack looks a bit different in the world of advanced analytics and AI.
The modern analytical stack is a highly disaggregated stack. Unlike traditional databases or data warehouses, the new stack is split apart.
- Pick a data lake or two or three to store data (S3, GCS, HDFS, etc.)
- Pick a computational framework to analyze data (Apache Spark, Presto, Hive, TensorFlow etc.)
- Make sure all the other dependencies like the catalog service are available (Hive Metastore, AWS Glue, KMS, etc.)
Challenges running the disaggregated analytics stack in K8s
Kubernetes greatly simplifies the complexity of deploying so many distributed systems together. And overtime advanced analytics running on K8s clusters will become the norm. But there are still a few critical gaps to make this modern analytical stack effective.
Challenge #1 - No shared data access / caching layer in the K8s cluster
K8s is a fantastic container orchestration technology and with tools like Helm charts, operators and more, deployment can be greatly simplified. However, for data intensive workloads like advanced analytics typically need data sharing between jobs to be effective so that data from one job can be easily be accessed by the next job. Without a data access / caching layer, the data needs to be written back to the data lake and then needs to be read back into the K8s cluster again significantly slowing down data pipelines.
Challenge #2 - Lost data locality
With data being stored in S3 or other cloud object stores or on-prem in Hadoop, to perform analytics within the K8s cluster, users have a couple of options. Data needs to either be accessed remotely (meaning poor performance) or needs to be manually copied into the K8s cluster (meaning a lot more additional DevOps and management on a per workload basis). And oftentimes this will carry the burden of managing the differences between those copies which can be hard. The ideal solution is for data locality to be recreated in this disaggregated stack.
Challenge #3 - No data elasticity for elastic compute
The beauty of K8s is the flexibility it gets to even the most complex compute workloads - scale up, down, upgrade, restart etc. based on need and demand. But again the dependency on data being available to compute remains for data-intensive workloads. To scale compute in, out, up or down, the data within K8s also needs to be able to do the same to leverage the power of the flexibility K8s brings.
Data Orchestration can solve these challenges by syncing data into the K8s cluster and allowing for seamless in-memory data access and flexibility to share data across jobs and scale in or out as needed.
The news
Alluxio has had a docker container for a while, but with Alluxio version 2.1, Kubernetes becomes a first class environment for Alluxio with advanced testing and certification of K8s. We are now seeing more production deployments with Alluxio and compute frameworks like Presto and Spark in K8s.
Also new with Alluxio version 2.1, Alluxio is available for deployment via Helm Charts.
What are Helm Charts?
Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste. Learn more here: https://helm.sh
Get started
- To learn more about deploying Alluxio with Helm Charts, read the docs.
- You can get started with trying out Alluxio using our docker sandbox tutorial!
.png)
Blog
We are thrilled to announce the general availability of Alluxio Enterprise for Data Analytics 3.2! With data volumes continuing to grow at exponential rates, data platform teams face challenges in maintaining query performance, managing infrastructure costs, and ensuring scalability. This latest version of Alluxio addresses these challenges head-on with groundbreaking improvements in scalability, performance, and cost-efficiency.
.jpeg)
We’re excited to introduce Rapid Alluxio Deployer (RAD) on AWS, which allows you to experience the performance benefits of Alluxio in less than 30 minutes. RAD is designed with a split-plane architecture, which ensures that your data remains secure within your AWS environment, giving you peace of mind while leveraging Alluxio’s capabilities.
PyTorch is one of the most popular deep learning frameworks in production today. As models become increasingly complex and dataset sizes grow, optimizing model training performance becomes crucial to reduce training times and improve productivity.