HyperJoin: LLM-augmented Hypergraph Link Prediction for Joinable Table Discovery

TL;DR

HyperJoin tackles joinable table discovery in data lakes by recasting the task as link prediction on a dual-type hypergraph. It jointly models intra-table structure and inter-table joinability using LLM-augmented hyperedges, and learns expressive column representations with a Hierarchical Interaction Network (HIN) that performs multi-level message passing. Online ranking is made coherent and scalable through a two-stage process that retrieves a candidate pool and applies an MST-based greedy reranking to maximize global coherence, with theoretical guarantees and NP-hardness justification for the coherence objective. Empirically, HyperJoin outperforms state-of-the-art baselines across multiple benchmarks, delivering substantial gains in Precision@K and Recall@K while maintaining efficiency, and demonstrating robust performance under ablations and parameter variations.

Abstract

As a pivotal task in data lake management, joinable table discovery has attracted widespread interest. While existing language model-based methods achieve remarkable performance by combining offline column representation learning with online ranking, their design insufficiently accounts for the underlying structural interactions: (1) offline, they directly model tables into isolated or pairwise columns, thereby struggling to capture the rich inter-table and intra-table structural information; and (2) online, they rank candidate columns based solely on query-candidate similarity, ignoring the mutual interactions among the candidates, leading to incoherent result sets. To address these limitations, we propose HyperJoin, a large language model (LLM)-augmented Hypergraph framework for Joinable table discovery. Specifically, we first construct a hypergraph to model tables using both the intra-table hyperedges and the LLM-augmented inter-table hyperedges. Consequently, the task of joinable table discovery is formulated as link prediction on this constructed hypergraph. We then design HIN, a Hierarchical Interaction Network that learns expressive column representations through bidirectional message passing over columns and hyperedges. To strengthen coherence and internal consistency in the result columns, we cast online ranking as a coherence-aware top-k column selection problem. We then introduce a reranking module that leverages a maximum spanning tree algorithm to prune noisy connections and maximize coherence. Experiments demonstrate the superiority of HyperJoin, achieving average improvements of 21.4% (Precision@15) and 17.2% (Recall@15) over the best baseline.

Figures (6)

• Figure 1: An example of joinable table discovery in data lake.
• Figure 2: Framework comparisons of joinable table discovery methods.
• Figure 3: Framework overview of HyperJoin.
• Figure 4: Performance comparison across different datasets and k values.
• Figure 5: Training and evaluation efficiency across datasets.

Theorems & Definitions (16)

• Example 1
• Definition 1: Pairwise Column Joinability
• Definition 2: Joinable Table Discovery
• Theorem 1
• Theorem 2
• Theorem 3: is strictly more expressive than 1-WL message passing
• Theorem 4: NP-hardness of Coherence-Aware Selection
• Definition 3: Candidate Graph
• Lemma 1: MST Gain Lower Bound
• Theorem 5: Surrogate Marginal Gain Property