Alluxio for Presto fu can across the cloud self-service ability

Table of Contents

What kind of architecture is self-service capable?

Considerations for designing a data platform

Conclusion

---

This article highlights the synergy between two popular open source projects, Alluxio and Presto, and shows how to leverage both to enable cross-cloud self-service data architectures.

about the author
Fan BinAlluxio VP Open Source and Founding Member
Adit MadanAlluxio Senior Product Manager
Jasmine WangAlluxio Community Manager

What kind of architecture is self-service capable?

Let's discuss a question first, what conditions are met for this architecture to be called self-service.

Condition 1: As the data platform is updated, the architecture does not need to be modified

All data platforms evolve over time, including adding new data stores, computing engines, or having new teams that need access to shared data.In either case, such a platform is capable of self-service if these changes do not require modifications to the existing architecture.

Condition 2: Data isolation across teams

With a self-service platform, business units don't interfere with each other.When a new team joins, data can be shared, and the new data access will not affect the use of the original platform.

Agility is achieved if the above two conditions are met.When designing an architecture, it is more important to consider the ability to enable self-service than the cost of the physical architecture.

Considerations for designing a data platform

Below, we describe some of the considerations when designing a self-service platform, along with simplified architectural patterns and solutions.

Consideration 1: Data is shared

Share data between different computing frameworks

• Enterprises use various computing engines in the data platform, and each engine completes a specific task. For example, ETL batch processing is performed first, and then Presto is used for interactive query.This means that data is shared between different engines and between different teams
• For example, a team is responsible for collecting business data and sharing the data for use by multiple business units

Data centers across regions and data sharing across cloud vendors

• This allows the flexibility to choose the optimal storage environment and cloud service

How to solve the problem of data sharing, we propose the concept of an abstraction layer, and use the abstraction layer to realize heterogeneous computing across cross-environments.Alluxio provides such a cross-cloud abstraction layer, enabling seamless data sharing between Presto and other computing engines, no matter where the data is stored.

Consideration 2: The data has a business domain to which it belongs, the easiest way is to leave it in place

• Although copying can achieve data isolation, when the data access policy is very strict, the use of data by the data producer needs to be strictly controlled, and the entire data governance will become very complicated.
• Data copying leads to redundant storage space, is prone to errors, and takes up a lot of resources.

Copying data is obviously not an ideal solution, but how to achieve high performance for heterogeneous data access without moving the data?This requires abstraction layers to address data governance, performance, and moving data across businesses.

The architecture below shows how Presto utilizes Alluxio as an abstraction layer to access data located in different storage environments.

Generally, there are two situations:

• All data in a single cloud or single data center
• Data is shared across multiple data centers or hybrid clouds

In either case, Alluxio acts as an abstraction layer to isolate data consumers and producers.The abstraction layer is not just used as a cache, the ability to preload and write in advance ensures that the SLA is consistent even when the data is separated from the calculation.

Conclusion

Alluxio empowers Presto with self-service capabilities. Through Alluxio, a cross-cloud self-service data architecture can be realized, and the entire architecture can better adapt to the evolution of the data platform.If you want to know more information, you can check the white paper《Alluxio+Presto概述——Architecture Evolution of Interactive Queries" to learn how Facebook, TikTok, Electronic Arts, Walmart, Tencent, Comcast and more are using Alluxio to optimize the Presto platform.