Resource Hub

This is a blog series talking about the design and implementation of the Cross Cluster Synchronization mechanism in Alluxio. This mechanism ensures that the metadata is consistent when running multiple Alluxio clusters. Part 1 of this blog series discusses the scenario and background.

Following part 1 and part 2, this final blog of the series discusses some design decisions and details, as well as certain future work.

This is part 2 of the blog series talking about the design and implementation of the Cross Cluster Synchronization mechanism in Alluxio. In the previous blog, we discussed the scenario, background and how metadata sync is done with a single Alluxio cluster. This blog will describe how metadata sync is built upon to provide metadata consistency in a multi-cluster scenario.

‍

Maximizing Efficiency and Reducing S3 Egress Cost with Hybrid Cloud Data Access

Modern analytics projects rely on a hodgepodge of compute clusters, data stores, and pipelines, flung across countries and continents. Enterprises struggle to meet performance SLAs without replicating lots of data or moving and re-coding applications.

‍

Big Data Bellevue Meetup

Big Data Bellevue & Cloudy With a Chance of Data Meetup

A Fortune 50 technology company that serves over 1 billion users successfully implemented Alluxio to achieve a hybrid cloud strategy, become multi-cloud ready, cut costs, and boost agility.

Unify Data Lakes Across Multiple Geographic Regions in the Cloud

The problem with data modernization initiatives is that they result in distributed datasets that impede analytics projects. As enterprises start their cloud migration journey, adopt new types of applications, data stores, and infrastructure, they still leave residual data in the original location. This results in far-flung silos that can be slow, complex and expensive to analyze. As business demands for analytics rise—along with cloud costs—enterprises need to rationalize how they access and process distributed data. They cannot afford to replicate entire datasets or rewrite software every time they study data in more than one location.

‍

ALLUXIO DAY x APAC Modern Data Stack 2022

Alluxio (www.alluxio.io) is an open-source virtual distributed file system that provides a unified data access layer for hybrid and multi-cloud deployments. It enables distributed compute engines like Spark, Presto or Machine Learning frameworks like TensorFlow to transparently access different persistent storage systems (including HDFS, S3, Azure and etc) while actively leveraging in-memory cache to accelerate data access. Developed originally from UC Berkeley AMPLab as research project “Tachyon”, Alluxio has more than 1200 contributors and is used by over 100 companies worldwide with the largest production deployment over 1000 nodes.

ALLUXIO DAY x APAC Modern Data Stack 2022

ALLUXIO DAY x APAC Modern Data Stack 2022

‍

ALLUXIO DAY x APAC Modern Data Stack 2022

‍