QJoin: Transformation-aware Joinable Data Discovery Using Reinforcement Learning
Ning Wang, Sainyam Galhotra
TL;DR
QJoin addresses the challenge of join discovery across heterogeneous data repositories by learning transformation sequences with reinforcement learning and reusing successful patterns across tasks. It introduces a UNIQ-ALCS reward that balances alignment quality with key distinctiveness, and two reuse mechanisms: agent transfer and a Transformation Reuse Library. Through a staged pipeline—pre-scoring, clustering, cluster-specific RL, and adaptive join execution—QJoin achieves high F1 on AutoJoin Web (~91%) and reduces runtime on NYC+Chicago tasks by up to ~7.4%. These results show that transformation-learning plus reuse substantially improves both accuracy and efficiency for join discovery in large-scale, open data settings.
Abstract
Discovering which tables in large, heterogeneous repositories can be joined and by what transformations is a central challenge in data integration and data discovery. Traditional join discovery methods are largely designed for equi-joins, which assume that join keys match exactly or nearly so. These techniques, while efficient in clean, well-normalized databases, fail in open or federated settings where identifiers are inconsistently formatted, embedded, or split across multiple columns. Approximate or fuzzy joins alleviate minor string variations but cannot capture systematic transformations. We introduce QJoin, a reinforcement-learning framework that learns and reuses transformation strategies across join tasks. QJoin trains an agent under a uniqueness-aware reward that balances similarity with key distinctiveness, enabling it to explore concise, high-value transformation chains. To accelerate new joins, we introduce two reuse mechanisms: (i) agent transfer, which initializes new policies from pretrained agents, and (ii) transformation reuse, which caches successful operator sequences for similar column clusters. On the AutoJoin Web benchmark (31 table pairs), QJoin achieves an average F1-score of 91.0%. For 19,990 join tasks in NYC+Chicago open datasets, Qjoin reduces runtime by up to 7.4% (13,747 s) by using reusing. These results demonstrate that transformation learning and reuse can make join discovery both more accurate and more efficient.