Fuzzy Integration of Data Lake Tables
Aamod Khatiwada, Roee Shraga, Renée J. Miller
TL;DR
The paper tackles data lake table integration under inconsistent values and unreliable metadata by introducing Fuzzy Full Disjunction, which extends the associative Full Disjunction operator to handle approximate matches. It first performs a value-matching step by embedding column values with models like Mistral-7B-Instruct, measuring cosine distances, and clustering values via a threshold $\theta$, before collapsing matches into representative values and applying the standard $FD$ operator. The approach is evaluated on Auto-Join and ALITE benchmarks, showing improved downstream entity resolution (e.g., $P$, $R$, $F1$ gains) with negligible runtime overhead compared to equi-join FD baselines. This work enables more effective and scalable data-lake integration by accommodating synonyms, abbreviations, and other inconsistencies in join values, thereby improving the quality of integrated views for downstream analytics.
Abstract
Data integration is an important step in any data science pipeline where the objective is to unify the information available in different datasets for comprehensive analysis. Full Disjunction, which is an associative extension of the outer join operator, has been shown to be an effective operator for integrating datasets. It fully preserves and combines the available information. Existing Full Disjunction algorithms only consider the equi-join scenario where only tuples having the same value on joining columns are integrated. This, however, does not realistically represent an open data scenario, where datasets come from diverse sources with inconsistent values (e.g., synonyms, abbreviations, etc.) and with limited metadata. So, joining just on equal values severely limits the ability of Full Disjunction to fully combine datasets. Thus, in this work, we propose an extension of Full Disjunction to also account for "fuzzy" matches among tuples. We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction. Experimentally, we show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation and also that it enhances the integration effectiveness.