Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XTable in Action: Seamless Interoperability in Data Lakes

Ashvin Agrawal, Tim Brown, Anoop Johnson, Jesús Camacho-Rodríguez, Kyle Weller, Carlo Curino, Raghu Ramakrishnan

TL;DR

The paper addresses data estate fragmentation from multiple open LST formats (Delta Lake, Iceberg, Hudi) by introducing XTable, an omni-directional, low-overhead translator that enables writing data in one LST and reading it in others. XTable relies on a universal internal representation and a three-component architecture (Source Readers, Target Writers, Core Logic) to perform metadata-focused, incremental translations asynchronously, without duplicating data or requiring engine changes. It extends beyond existing one-way translators by supporting multi-directional interoperability and extensibility to new formats. The demonstration showcases realistic scenarios where cross-format access, multi-engine analytics, and performance-driven format choices are achieved without data replication, enabling unified analytics in data lakes.

Abstract

Contemporary approaches to data management are increasingly relying on unified analytics and AI platforms to foster collaboration, interoperability, seamless access to reliable data, and high performance. Data Lakes featuring open standard table formats such as Delta Lake, Apache Hudi, and Apache Iceberg are central components of these data architectures. Choosing the right format for managing a table is crucial for achieving the objectives mentioned above. The challenge lies in selecting the best format, a task that is onerous and can yield temporary results, as the ideal choice may shift over time with data growth, evolving workloads, and the competitive development of table formats and processing engines. Moreover, restricting data access to a single format can hinder data sharing resulting in diminished business value over the long term. The ability to seamlessly interoperate between formats and with negligible overhead can effectively address these challenges. Our solution in this direction is an innovative omni-directional translator, XTable, that facilitates writing data in one format and reading it in any format, thus achieving the desired format interoperability. In this work, we demonstrate the effectiveness of XTable through application scenarios inspired by real-world use cases.

XTable in Action: Seamless Interoperability in Data Lakes

TL;DR

The paper addresses data estate fragmentation from multiple open LST formats (Delta Lake, Iceberg, Hudi) by introducing XTable, an omni-directional, low-overhead translator that enables writing data in one LST and reading it in others. XTable relies on a universal internal representation and a three-component architecture (Source Readers, Target Writers, Core Logic) to perform metadata-focused, incremental translations asynchronously, without duplicating data or requiring engine changes. It extends beyond existing one-way translators by supporting multi-directional interoperability and extensibility to new formats. The demonstration showcases realistic scenarios where cross-format access, multi-engine analytics, and performance-driven format choices are achieved without data replication, enabling unified analytics in data lakes.

Abstract

Contemporary approaches to data management are increasingly relying on unified analytics and AI platforms to foster collaboration, interoperability, seamless access to reliable data, and high performance. Data Lakes featuring open standard table formats such as Delta Lake, Apache Hudi, and Apache Iceberg are central components of these data architectures. Choosing the right format for managing a table is crucial for achieving the objectives mentioned above. The challenge lies in selecting the best format, a task that is onerous and can yield temporary results, as the ideal choice may shift over time with data growth, evolving workloads, and the competitive development of table formats and processing engines. Moreover, restricting data access to a single format can hinder data sharing resulting in diminished business value over the long term. The ability to seamlessly interoperate between formats and with negligible overhead can effectively address these challenges. Our solution in this direction is an innovative omni-directional translator, XTable, that facilitates writing data in one format and reading it in any format, thus achieving the desired format interoperability. In this work, we demonstrate the effectiveness of XTable through application scenarios inspired by real-world use cases.
Paper Structure (6 sections, 3 figures)

This paper contains 6 sections, 3 figures.

Table of Contents

  1. Introduction
  2. LST Overview
  3. XTable Overview
  4. XTable Architecture
  5. Related Work
  6. Demonstration Overview

Figures (3)

  • Figure 1: Simplified layout of metadata files and referenced data files for example in \ref{['lst:iceberg_sample']}
  • Figure 2: Overview of the XTable Architecture.
  • Figure 3: Scenario 2: Team-A uses Iceberg exclusively, and Team B uses Hudi. XTable enables access to Stocks and Crypto tables in both formats for seamless interop.