Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method

Xiyang Zhang, Chen Liang, Haoxuan Qiu, Hongzhi Wang

TL;DR

Data selection for MLPs across multiple heterogeneous sources under budget constraints is challenging due to evolving representations and non-convex training dynamics. The authors introduce Data Value Contribution (DVC), which decomposes sample value into Layer Value Contribution (LVC) and Global Value Contribution (GVC) evaluated by six metrics, and guides adaptive source selection with a UCB-based bandit. They establish sublinear complexity $O(\text{budget} \cdot n^{\rho} \log n \cdot d)$ with $\rho < 1$, and provide gradient caching, LSH-based similarity, online statistics, and Bayesian-calibrated weighting to support scalability and theoretical guarantees. Empirically, DVC outperforms eight baselines across eight datasets and budgets, with strong performance under noise and distribution shifts and clear scalability advantages. This work delivers a principled, scalable framework for hierarchical data valuation in large-scale MLP training with practical impact for production systems.

Abstract

Data selection is one of the fundamental problems in neural network training, particularly for multi-layer perceptrons (MLPs) where identifying the most valuable training samples from massive, multi-source, and heterogeneous data sources under budget constraints poses significant challenges. Existing data selection methods, including coreset construction, data Shapley values, and influence functions, suffer from critical limitations: they oversimplify nonlinear transformations, ignore informative intermediate representations in hidden layers, or fail to scale to larger MLPs due to high computational complexity. In response, we propose DVC (Data Value Contribution), a novel budget-aware method for evaluating and selecting data for MLP training that accounts for the dynamic evolution of network parameters during training. The DVC method decomposes data contribution into Layer Value Contribution (LVC) and Global Value Contribution (GVC), employing six carefully designed metrics and corresponding efficient algorithms to capture data characteristics across three dimensions--quality, relevance, and distributional diversity--at different granularities. DVC integrates these assessments with an Upper Confidence Bound (UCB) algorithm for adaptive source selection that balances exploration and exploitation. Extensive experiments across six datasets and eight baselines demonstrate that our method consistently outperforms existing approaches under various budget constraints, achieving superior accuracy and F1 scores. Our approach represents the first systematic treatment of hierarchical data evaluation for neural networks, providing both theoretical guarantees and practical advantages for large-scale machine learning systems.

Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method

TL;DR

Data selection for MLPs across multiple heterogeneous sources under budget constraints is challenging due to evolving representations and non-convex training dynamics. The authors introduce Data Value Contribution (DVC), which decomposes sample value into Layer Value Contribution (LVC) and Global Value Contribution (GVC) evaluated by six metrics, and guides adaptive source selection with a UCB-based bandit. They establish sublinear complexity with , and provide gradient caching, LSH-based similarity, online statistics, and Bayesian-calibrated weighting to support scalability and theoretical guarantees. Empirically, DVC outperforms eight baselines across eight datasets and budgets, with strong performance under noise and distribution shifts and clear scalability advantages. This work delivers a principled, scalable framework for hierarchical data valuation in large-scale MLP training with practical impact for production systems.

Abstract

Data selection is one of the fundamental problems in neural network training, particularly for multi-layer perceptrons (MLPs) where identifying the most valuable training samples from massive, multi-source, and heterogeneous data sources under budget constraints poses significant challenges. Existing data selection methods, including coreset construction, data Shapley values, and influence functions, suffer from critical limitations: they oversimplify nonlinear transformations, ignore informative intermediate representations in hidden layers, or fail to scale to larger MLPs due to high computational complexity. In response, we propose DVC (Data Value Contribution), a novel budget-aware method for evaluating and selecting data for MLP training that accounts for the dynamic evolution of network parameters during training. The DVC method decomposes data contribution into Layer Value Contribution (LVC) and Global Value Contribution (GVC), employing six carefully designed metrics and corresponding efficient algorithms to capture data characteristics across three dimensions--quality, relevance, and distributional diversity--at different granularities. DVC integrates these assessments with an Upper Confidence Bound (UCB) algorithm for adaptive source selection that balances exploration and exploitation. Extensive experiments across six datasets and eight baselines demonstrate that our method consistently outperforms existing approaches under various budget constraints, achieving superior accuracy and F1 scores. Our approach represents the first systematic treatment of hierarchical data evaluation for neural networks, providing both theoretical guarantees and practical advantages for large-scale machine learning systems.
Paper Structure (36 sections, 2 theorems, 15 equations, 2 figures, 3 tables, 5 algorithms)

This paper contains 36 sections, 2 theorems, 15 equations, 2 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Uncertainty-Based Data Selection
  4. Core-Set-Based Data Selection
  5. Shapley-Based Data Selection
  6. Influence-Based Data Selection
  7. Meta-Learning-Based Data Selection
  8. Overview
  9. Problem Definition
  10. Problem Analysis
  11. Method Overview
  12. Methodology
  13. Data Value Contribution Evaluation Metrics
  14. Layer-wise Value Metrics
  15. Quality Assessment $Q_l(x)$
  16. ...and 21 more sections

Key Result

theorem 1

Consider the UCB-based source selection algorithm where each source $i \in \{1, 2, \ldots, K\}$ has an unknown expected reward $\mu_i$, and rewards are bounded in $[0, 1]$. Let $\mu^* = \max_i \mu_i$ and $\Delta_i = \mu^* - \mu_i$ be the suboptimality gap of source $i$. Then the expected cumulative

Figures (2)

  • Figure 1: A Financial Fraud Detection System Example
  • Figure 2: DVC Method Workflow: Three Phases, Six Metrics and Five Algorithms included

Theorems & Definitions (2)

  • theorem 1: UCB Regret Bound for DVC Source Selection
  • theorem 2: Comprehensive Complexity Analysis