LakeVilla: A Modular and Non-Invasive Toolbox for Lakehouse Transactions

TL;DR

LakeVilla introduces a modular, built-in extension to Open Table Formats that enables recovery, complex multi-table transactions, and isolation in Lakehouses without rearchitecting the underlying OTFs. By combining marker-based recovery (LV[R]), cross-table conflict detection with marker shifts (LV[CT]), and a global version log with validation (LV[I]), LakeVilla achieves strong ACID guarantees across multi-table and long-running workloads while incurring only modest overhead on typical LH workloads. The approach preserves compatibility with existing formats like Delta Lake and Iceberg and avoids vendor lock-in by operating directly within the OTF metadata layer. Experimental results show low throughput impact and clear improvements in consistency and abort handling for concurrent, multi-table transactions, making LakeVilla a practical path to scalable, transactional lakehouse systems. Overall, LakeVilla demonstrates that a modular, non-invasive augmentation of OTFs can deliver serializability and robust multi-table semantics in cloud analytics stacks.

Abstract

Data lakehouses (LHs) are at the core of current cloud analytics stacks by providing elastic, relational compute on data in cloud data lakes across vendors. For relational semantics, they rely on open table formats (OTFs). Unfortunately, they have many missing features inherent to their metadata designs, like no support for multi-table transactions and recovery in case of an abort in concurrent, multi-query workloads. This, in turn, can lead to non-repeatable reads, stale data, and high costs in production cloud systems. In this work, we introduce LakeVilla, a modular toolbox that introduces recovery, complex transactions, and transaction isolation to state-of-the-art OTFs like Apache Iceberg and Delta Lake tables. We investigate its transactional guarantees and show it has minimal impact on performance (2% YCSB writes, 2.5% TPC-DS reads) and provides concurrency control for multiple readers and writers for arbitrary long transactions in OTFs in a non-invasive way.

Figures (14)

• Figure 1: Metadata Layer design of Delta Lake in a banking example (running example) (draw.io)
• Figure 2: Overview of all LakeVilla (LV) features and how they integrate into the original OTF (draw.io)
• Figure 3: Module #1: Markers (green) reserve snapshot versions in the metadata layers (draw.io)
• Figure 4: Module #2: Logical and physical Sublog instructions for "UPDATE Bank x SET Balance=100 WHERE key=Acc. 2" (draw.io)
• Figure 5: Module #3: A marker-based conflict and deadlock detection model. This figure shows a conflict between tables 1 and 2 and how T1 recovers from it (draw.io)

Theorems & Definitions (13)

• definition thmcounterdefinition
• definition thmcounterdefinition
• lemma thmcounterlemma
• definition thmcounterdefinition
• lemma thmcounterlemma
• lemma thmcounterlemma
• lemma thmcounterlemma
• lemma thmcounterlemma
• lemma thmcounterlemma
• lemma thmcounterlemma