ShapleyPipe: Hierarchical Shapley Search for Data Preparation Pipeline Construction

TL;DR

ShapleyPipe reframes data preparation pipeline construction as a cooperative game, enabling principled, position-aware operator valuation via Shapley values and overcoming combinatorial explosion with a two-stage hierarchical search. Stage 1 uses a Multi-Armed Bandit to select category structures, while Stage 2 refines within-category operators through constrained Permutation Shapley evaluation, yielding polynomial-time complexity and substantial efficiency gains. Across 18 DiffPrep datasets, ShapleyPipe achieves an average accuracy of $0.835$, beating state-of-the-art RL methods, and approaches the high-budget upper bound while using significantly fewer evaluations; it also delivers interpretable operator valuations with strong empirical support ($\rho = 0.933$ between Shapley values and empirical performance). The approach demonstrates robust generalization, concrete theoretical guarantees (category coherence and estimation unbiasedness), and practical utility for data-driven library refinement and pipeline analysis. This framework advances automated data preparation by delivering both performance and interpretability, with potential for transfer learning and broader modality extensions.

Abstract

Automated data preparation pipeline construction is critical for machine learning success, yet existing methods suffer from two fundamental limitations: they treat pipeline construction as black-box optimization without quantifying individual operator contributions, and they struggle with the combinatorial explosion of the search space ($N^M$ configurations for N operators and pipeline length M). We introduce ShapleyPipe, a principled framework that leverages game-theoretic Shapley values to systematically quantify each operator's marginal contribution while maintaining full interpretability. Our key innovation is a hierarchical decomposition that separates category-level structure search from operator-level refinement, reducing the search complexity from exponential to polynomial. To make Shapley computation tractable, we develop: (1) a Multi-Armed Bandit mechanism for intelligent category evaluation with provable convergence guarantees, and (2) Permutation Shapley values to correctly capture position-dependent operator interactions. Extensive evaluation on 18 diverse datasets demonstrates that ShapleyPipe achieves 98.1\% of high-budget baseline performance while using 24\% fewer evaluations, and outperforms the state-of-the-art reinforcement learning method by 3.6\%. Beyond performance gains, ShapleyPipe provides interpretable operator valuations ($ρ$=0.933 correlation with empirical performance) that enable data-driven pipeline analysis and systematic operator library refinement.

Figures (6)

• Figure 1: Motivating examples showing operator selection and ordering both matter. (a) On Avila dataset, CtxPipe (0.759) vs. optimal (0.941) shows 24% gap. (b) Operator choice matters: Using PowerTransformer (PT, sensitive to outliers) vs. QuantileTransformer (QT, robust to outliers) at position 2 yields 18.2% difference. (c) On Pbcseq dataset, ordering matters: identical operators {One-Hot Encoding (OHE), Polynomial Features (PF), Robust Scaler (RBS)} achieve 0.743 vs 0.717 accuracy (3.6% gap).
• Figure 2: Correlation heatmap shows strong within-category coherence (diagonal blocks, $\rho_{\text{within}} = 0.310 \pm 0.249$) and weak between-category correlation (off-diagonal regions, $\rho_{\text{between}} = -0.098 \pm 0.235$), validating that operators partition into behaviorally distinct functional groups.
• Figure 3: Parallel Scalability on ShapleyPipe
• Figure 4: Accuracy-runtime Pareto frontier. ShapleyPipe (green star) achieves the highest accuracy (0.835) among methods with runtime under 1,000 seconds. Methods in the shaded region are strictly dominated. CtxPipe* excludes 36-hour training overhead.
• Figure 5: Accuracy convergence pattern with increasing permutation sampling size, averaged across 18 datasets. Our default $n_{\text{perm}}=75$ (purple star) achieves peak accuracy of 0.835. Notably, further increasing to $n_{\text{perm}}=100$ causes a 0.6% performance drop (red triangle), demonstrating that excessive sampling can degrade performance through overfitting to sampling-induced noise.

Theorems & Definitions (16)

• Definition 1: Data Preparation Operator
• Definition 2: Data Preparation Pipeline
• Definition 3: Performance Function
• Definition 4: Cooperative Game
• Definition 5: Shapley Value
• Definition 6: Conditional Position-Specific Shapley
• Theorem 1: Complexity of Position-Aware Shapley Construction
• proof
• Proposition 1: Search Space Factorization
• Definition 7: Category Coherence