EAIFD: A Fast and Scalable Algorithm for Incremental Functional Dependency Discovery

TL;DR

EAIFD addresses the problem of discovering minimal, non-trivial FDs under incremental updates by reframing FD discovery as minimal hitting-set enumeration over partial hypergraphs built from difference sets. It combines a sampling-based one-time initialization with a two-step incremental validation strategy that leverages a compact multi-attribute hash table (MHT) to prune candidates and minimize I/O, plus selective, block-wise validation via IGHV. The key innovations are the MHT structure with a high-frequency mapping preservation threshold and the iterative hypergraph refinement that avoids full re-runs, yielding up to an order-of-magnitude speedup and more than two orders of magnitude memory reduction compared to prior incremental approaches. Empirical results on real-world datasets demonstrate robust performance across varying dataset sizes, attribute counts, and incremental data proportions, supporting EAIFD’s scalability and practicality for continuous FD discovery in large databases.

Abstract

Functional dependencies (FDs) are fundamental integrity constraints in relational databases, but discovering them under incremental updates remains challenging. While static algorithms are inefficient due to full re-execution, incremental algorithms suffer from severe performance and memory bottlenecks. To address these challenges, this paper proposes EAIFD, a novel algorithm for incremental FD discovery. EAIFD maintains the partial hypergraph of difference sets and reframes the incremental FD discovery problem into minimal hitting set enumeration on hypergraph, avoiding full re-runs. EAIFD introduces two key innovations. First, a multi-attribute hash table ($MHT$) is devised for high-frequency key-value mappings of valid FDs, whose memory consumption is proven to be independent of the dataset size. Second, two-step validation strategy is developed to efficiently validate the enumerated candidates, which leverages $MHT$ to effectively reduce the validation space and then selectively loads data blocks for batch validation of remaining candidates, effectively avoiding repeated I/O operations. Experimental results on real-world datasets demonstrate the significant advantages of EAIFD. Compared to existing algorithms, EAIFD achieves up to an order-of-magnitude speedup in runtime while reducing memory usage by over two orders-of-magnitude, establishing it as a highly efficient and scalable solution for incremental FD discovery.

Figures (6)

• Figure 1: The figure of two student information tables. Table $1$ contains $6$ student information tuples, Table $2$ contains $4$ student information tuples, and the tuples in Table $2$ are incrementally added to Table $1$.
• Figure 2: Overview of EAIFD and its components.
• Figure 3: The process of IGHV. After grouping, select some $\mathcal{C}_{FDs}$ in $g_2$ and show its core validation steps on data blocks $r_1$ and $r_4$
• Figure 4: The process of validating $\mathcal{C}_{FDs}$ in two types: $\mathcal{C}_1$ and $\mathcal{C}_2$. For clarity, $\Delta r$ contains only 3 tuples, and two candidate FDs are validated in each type.
• Figure 5: Time performance of EAIFD compared with DynFD, and DHSFD on the CAB and Pitches. Subfigures (a, b, c) correspond to CAB with varying tuple number $|r|$, attribute number $|R|$, and incremental data size $|\Delta r|$, respectively, while (d, e, f) show the same for Pitches.

Theorems & Definitions (8)

• Example 1.1
• Definition 3.1
• Definition 3.2
• Definition 3.3
• Definition 3.4
• Lemma 4.1
• Corollary 4.1
• Corollary 4.2