Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Past Still Matters: A Temporally-Valid Data Discovery System

Mahdi Esmailoghli, Matthias Weidlich

TL;DR

Temporally drifted data discovery in data lakes lacking explicit temporal metadata is addressed by a two-phase system that splits queries into $Q_D$ (dataset) and $Q_V$ (version) components and leverages a temporally-enriched index. The offline phase builds version families, representative tables, temporal lineage, change logs, and a hierarchical three-level index, while the online phase performs query decoupling and time-travel-based reconstruction to retrieve temporally-valid results. Major contributions include a ML-based version-family detector, representative tables, unified change logs, and a hierarchical index that enables fast, temporally-consistent discovery. This framework improves reproducibility and accuracy of downstream analytics by ensuring the selected data versions align with the task's temporal context.

Abstract

Over the past decade, the proliferation of public and enterprise data lakes has fueled intensive research into data discovery, aiming to identify the most relevant data from vast and complex corpora to support diverse user tasks. Significant progress has been made through the development of innovative index structures, similarity measures, and querying infrastructures. Despite these advances, a critical aspect remains overlooked: relevance is time-varying. Existing discovery methods largely ignore this temporal dimension, especially when explicit date/time metadata is missing. To fill this gap, we outline a vision for a data discovery system that incorporates the temporal dimension of data. Specifically, we define the problem of temporally-valid data discovery and argue that addressing it requires techniques for version discovery, temporal lineage inference, change log synthesis, and time-aware data discovery. We then present a system architecture to deliver these techniques, before we summarize research challenges and opportunities. As such, we lay the foundation for a new class of data discovery systems, transforming how we interact with evolving data lakes.

The Past Still Matters: A Temporally-Valid Data Discovery System

TL;DR

Temporally drifted data discovery in data lakes lacking explicit temporal metadata is addressed by a two-phase system that splits queries into (dataset) and (version) components and leverages a temporally-enriched index. The offline phase builds version families, representative tables, temporal lineage, change logs, and a hierarchical three-level index, while the online phase performs query decoupling and time-travel-based reconstruction to retrieve temporally-valid results. Major contributions include a ML-based version-family detector, representative tables, unified change logs, and a hierarchical index that enables fast, temporally-consistent discovery. This framework improves reproducibility and accuracy of downstream analytics by ensuring the selected data versions align with the task's temporal context.

Abstract

Over the past decade, the proliferation of public and enterprise data lakes has fueled intensive research into data discovery, aiming to identify the most relevant data from vast and complex corpora to support diverse user tasks. Significant progress has been made through the development of innovative index structures, similarity measures, and querying infrastructures. Despite these advances, a critical aspect remains overlooked: relevance is time-varying. Existing discovery methods largely ignore this temporal dimension, especially when explicit date/time metadata is missing. To fill this gap, we outline a vision for a data discovery system that incorporates the temporal dimension of data. Specifically, we define the problem of temporally-valid data discovery and argue that addressing it requires techniques for version discovery, temporal lineage inference, change log synthesis, and time-aware data discovery. We then present a system architecture to deliver these techniques, before we summarize research challenges and opportunities. As such, we lay the foundation for a new class of data discovery systems, transforming how we interact with evolving data lakes.
Paper Structure (23 sections, 3 figures)

This paper contains 23 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Discovery
  4. Temporal Data Management
  5. Data Lake Systems
  6. Traditional and Time Series Databases
  7. Problem Characterization
  8. Version Discovery
  9. Temporal Lineage Inference
  10. Change Log Synthesis
  11. Temporally-valid Data Discovery
  12. System Architecture
  13. Offline Phase
  14. Version Family Discovery
  15. Representative Table Generation
  16. ...and 8 more sections

Figures (3)

  • Figure 1: An example of a query and three joinable tables.
  • Figure 2: Running example with two version families.
  • Figure 3: System architecture.