Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology-Aware Subset Repair via Entropy-Guided Density and Graph Decomposition

Guoqi Zhao, Xixian Han, Xiaolong Wan

TL;DR

This work tackles subset repair under CFDs by introducing a topology-aware framework that unifies entropy-guided density (EntroCFDensity) with a conflict-degree penalty and graph-decomposition. Conflict detection is accelerated by inverted indexes and CFD grouping, and the global repair is transformed into independent local problems on connected components, solved via PPIS (efficient greedy) and MICO (mixed-integer programming with guarantees). The approach addresses homogeneous attribute weighting, density bias from dirty clusters, and computational cost, demonstrating improved repair accuracy and robustness while preserving high-quality data across large datasets. The contributions yield a practical, scalable pipeline for constraint-driven data cleaning with strong theoretical and empirical performance advantages.

Abstract

Subset repair is an important data cleaning technique that enforces integrity constraints by deleting a minimal number of conflicting tuples, yet multiple minimal repairs often exist. Density-based methods address this ambiguity by favoring repairs that preserve dense, high-quality data regions; however, their effectiveness is limited by density bias from dirty clusters, high computational cost, and uniform attribute weighting. We propose a topology-aware approximate subset repair framework based on a joint density-conflict penalty model. The framework integrates three key components. First, a two-layer conflict detection strategy combines attribute inverted indexes with CFD rule grouping to efficiently identify violations. Second, we introduce EntroCFDensity, a density metric that incorporates information entropy and CFD weights to dynamically adjust attribute importance and reduce homogeneity bias. Third, a conflict degree measure is defined to complement local density, enabling a topology-adaptive penalty mechanism with dynamic weight allocation guided by the coefficient of variation. The conflict graph is further decomposed into independent subgraphs, transforming global repair into tractable local subproblems. Based on this framework, we develop two algorithms: PPIS, a scalable heuristic, and MICO, a mixed-integer programming method with theoretical guarantees. Experimental results show that our approach improves repair accuracy and robustness while effectively preserving high-quality data.

Topology-Aware Subset Repair via Entropy-Guided Density and Graph Decomposition

TL;DR

This work tackles subset repair under CFDs by introducing a topology-aware framework that unifies entropy-guided density (EntroCFDensity) with a conflict-degree penalty and graph-decomposition. Conflict detection is accelerated by inverted indexes and CFD grouping, and the global repair is transformed into independent local problems on connected components, solved via PPIS (efficient greedy) and MICO (mixed-integer programming with guarantees). The approach addresses homogeneous attribute weighting, density bias from dirty clusters, and computational cost, demonstrating improved repair accuracy and robustness while preserving high-quality data across large datasets. The contributions yield a practical, scalable pipeline for constraint-driven data cleaning with strong theoretical and empirical performance advantages.

Abstract

Subset repair is an important data cleaning technique that enforces integrity constraints by deleting a minimal number of conflicting tuples, yet multiple minimal repairs often exist. Density-based methods address this ambiguity by favoring repairs that preserve dense, high-quality data regions; however, their effectiveness is limited by density bias from dirty clusters, high computational cost, and uniform attribute weighting. We propose a topology-aware approximate subset repair framework based on a joint density-conflict penalty model. The framework integrates three key components. First, a two-layer conflict detection strategy combines attribute inverted indexes with CFD rule grouping to efficiently identify violations. Second, we introduce EntroCFDensity, a density metric that incorporates information entropy and CFD weights to dynamically adjust attribute importance and reduce homogeneity bias. Third, a conflict degree measure is defined to complement local density, enabling a topology-adaptive penalty mechanism with dynamic weight allocation guided by the coefficient of variation. The conflict graph is further decomposed into independent subgraphs, transforming global repair into tractable local subproblems. Based on this framework, we develop two algorithms: PPIS, a scalable heuristic, and MICO, a mixed-integer programming method with theoretical guarantees. Experimental results show that our approach improves repair accuracy and robustness while effectively preserving high-quality data.
Paper Structure (19 sections, 1 theorem, 33 equations, 8 figures, 7 tables, 3 algorithms)

This paper contains 19 sections, 1 theorem, 33 equations, 8 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rule-based data cleaning
  4. ML-based data cleaning
  5. Probabilistic and statistical data cleaning
  6. Summary and Positioning
  7. Problem definition
  8. Conflict Modeling and Adaptive Penalty Mechanism
  9. Conflict Detection Based on Attribute Inverted Index and CFDs Grouping
  10. Calculation of Density-Conflict Degree Penalty Score
  11. Graph-Decomposition Based Approximate S-repair Algorithms
  12. Connected Component Decomposition of Conflict Graph
  13. Removal of tuples
  14. Approximate boundary analysis of MICO
  15. Performance evaluation
  16. ...and 4 more sections

Key Result

Lemma 5.1

The global minimal removal set $I_n$ can be decomposed as where $I_i$ is the local removal set of connected component $G_i$ and $m$ is the total number of connected components.

Figures (8)

  • Figure 1: Schematic diagram of CFDs grouping
  • Figure 2: Diagram of Conflict Detection Process
  • Figure 3: The conflict degree between dirty data and clean data in six datasets
  • Figure 4: Flowchart of Topology-Aware Subset Repair via Entropy-Guided Density and Graph Decomposition
  • Figure 5: Illustration of Dividing-and-Conquering Removal for clique components and PPIS for non-clique components
  • ...and 3 more figures

Theorems & Definitions (18)

  • Example 1.1
  • Definition 3.1: Functional Dependency (FD)
  • Example 3.1
  • Definition 3.2: Conditional Functional Dependency (CFD)
  • Example 3.2
  • Definition 3.3: Conflict Tuple
  • Definition 3.4: Conflict Graph
  • Definition 3.5: Connected Components of Conflict Graph
  • Definition 3.6: Clique Component
  • Definition 3.7: Non-Clique Component
  • ...and 8 more