Efficient Row-Level Lineage Leveraging Predicate Pushdown
Yin Lin, Cong Yan
TL;DR
PredTrace introduces a predicate-pushdown-based approach to row-level data lineage that works across complex pipelines and UDFs with minimal runtime overhead. It offers a precise lineage path when intermediate results can be materialized and a practical, non-materialized mode that yields lineage supersets with iterative refinement to reduce false positives. The framework leverages an external pushdown module and SMT verification to ensure correctness, and optimizes intermediate results to achieve up to 99% storage reduction and substantial lineage-query speedups. Evaluations on TPC-H and real-world Pandas pipelines show broad coverage, competitive overhead, and strong lineage-query performance relative to prior lazy and eager methods. PredTrace thus provides a versatile, scalable solution for debugging, auditing, and data governance in modern data processing environments.
Abstract
Row-level lineage explains what input rows produce an output row through a data processing pipeline, having many applications like data debugging, auditing, data integration, etc. Prior work on lineage falls in two lines: eager lineage tracking and lazy lineage inference. Eager tracking integrates lineage tracing tightly into the operator implementation, enabling efficient customized tracking. However, this approach is intrusive, system-specific, and lacks adaptability. In contrast, lazy inference generates additional queries to compute lineage; it can be easily applied to any database, but the lineage query is usually slow. Furthermore, both approaches have limited coverage of the type of data processing pipeline supported due to operator-specific tracking or inference rules. In this work, we propose PredTrace, a lineage inference approach that achieves easy adaptation, low runtime overhead, efficient lineage querying, and high pipeline coverage. It achieves this by leveraging predicate pushdown: pushing a row-selection predicate that describes the target output down to source tables and querying the lineage by running the pushed-down predicate. PredTrace may require saving intermediate results when running the pipeline in order to compute the precise lineage. When this is not viable, it can still infer lineage but may return a superset. Compared to prior work, PredTrace achieves higher coverage on TPC-H queries as well as 70 sampled real-world data processing pipelines in which UDFs are widely used. It can infer lineage in seconds, outperforming prior lazy approaches by up to 10x.