Redundancy-Driven Top-$k$ Functional Dependency Discovery

TL;DR

This work tackles the information overload of traditional FD discovery by proposing SDP, an exact top-$k$ FD discovery method guided by redundancy. SDP employs a monotone upper bound on redundancy to prune search branches and integrates three optimizations—heuristic attribute ordering, PCM-guided pruning, and a global best-first scheduler—to accelerate discovery while preserving optimality. Empirical results on 40+ real datasets show SDP delivering 10–1000× speedups and reduced memory usage compared with exhaustive baselines, particularly on high-dimensional or large-scale data; FDR can fail due to memory constraints on such data. The approach yields a compact, high-signal set of FDs with strong practical relevance for normalization and data cleaning, demonstrating the value of redundancy-driven selective discovery in FD mining.

Abstract

Functional dependencies (FDs) are basic constraints in relational databases and are used for many data management tasks. Most FD discovery algorithms find all valid dependencies, but this causes two problems. First, the computational cost is prohibitive: computational complexity grows quadratically with the number of tuples and exponentially with the number of attributes, making discovery slow on large-scale and high-dimensional data. Second, the result set can be huge, making it hard to identify useful dependencies. We propose SDP (Selective-Discovery-and-Prune), which discovers the top-$k$ FDs ranked by redundancy count. Redundancy count measures how much duplicated information an FD explains and connects directly to storage overhead and update anomalies. SDP uses an upper bound on redundancy to prune the search space. It is proved that this upper bound is monotone: adding attributes refines partitions and thus decreases the bound. Once the bound falls below the top-$k$ threshold, the entire branch can be skipped. We improve SDP with three optimizations: ordering attributes by partition cardinality, using pairwise statistics in a Partition Cardinality Matrix to tighten bounds, and a global scheduler to explore promising branches first. Experiments on over 40 datasets show that SDP is much faster and uses less memory than exhaustive methods.

Figures (5)

• Figure 1: A motivating example for redundancy of FDs
• Figure 2: The illustration of scalability analysis with tuple numbers (ftda data set).
• Figure 3: The illustration of scalability analysis with attribute numbers (superconductivity data set).
• Figure 4: The illustration of impact of k (ReactionNetwork data set).
• Figure 5: The illustration of threshold value for different optimization strategies.

Theorems & Definitions (18)

• Example 1
• Definition 1: Functional Dependency
• Definition 2: Redundancy Count
• Definition 3: Top-$k$ FD Discovery
• Theorem 5.1
• proof
• Theorem 5.2
• proof
• Theorem 5.3
• proof