Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

Kaiqi Zhang, Yu-Xiang Wang

TL;DR

The paper tackles adaptive function estimation in nonparametric regression under heterogeneous smoothness by analyzing parallel ReLU neural networks trained with standard weight decay. It proves that weight decay induces an effective $\ell_{2/L}$ sparsity on the learned dictionary, making deep parallel networks behave like sparse regression with representation learning. The main result shows that for targets in Besov or BV classes, the estimator can achieve near-minimax rates, with the error approaching the optimal rate as depth increases. This provides a theoretical explanation for the empirical depth advantage of DNNs and suggests a practical pathway to adaptivity across diverse smoothness regimes without tuning architecture to specific function spaces. The work bridges neural network theory with classical nonparametric estimation, highlighting both the synthesis (basis construction) and analysis (regularization) perspectives in a unified framework.

Abstract

We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to adaptively estimate functions with heterogeneous smoothness -- a property of functions in Besov or Bounded Variation (BV) classes. Existing work on this problem requires tuning the NN architecture based on the function spaces and sample size. We consider a "Parallel NN" variant of deep ReLU networks and show that the standard $\ell_2$ regularization is equivalent to promoting the $\ell_p$-sparsity ($0<p<1$) in the coefficient vector of an end-to-end learned function bases, i.e., a dictionary. Using this equivalence, we further establish that by tuning only the regularization factor, such parallel NN achieves an estimation error arbitrarily close to the minimax rates for both the Besov and BV classes. Notably, it gets exponentially closer to minimax optimal as the NN gets deeper. Our research sheds new lights on why depth matters and how NNs are more powerful than kernel methods.

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

TL;DR

The paper tackles adaptive function estimation in nonparametric regression under heterogeneous smoothness by analyzing parallel ReLU neural networks trained with standard weight decay. It proves that weight decay induces an effective sparsity on the learned dictionary, making deep parallel networks behave like sparse regression with representation learning. The main result shows that for targets in Besov or BV classes, the estimator can achieve near-minimax rates, with the error approaching the optimal rate as depth increases. This provides a theoretical explanation for the empirical depth advantage of DNNs and suggests a practical pathway to adaptivity across diverse smoothness regimes without tuning architecture to specific function spaces. The work bridges neural network theory with classical nonparametric estimation, highlighting both the synthesis (basis construction) and analysis (regularization) perspectives in a unified framework.

Abstract

We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to adaptively estimate functions with heterogeneous smoothness -- a property of functions in Besov or Bounded Variation (BV) classes. Existing work on this problem requires tuning the NN architecture based on the function spaces and sample size. We consider a "Parallel NN" variant of deep ReLU networks and show that the standard regularization is equivalent to promoting the -sparsity () in the coefficient vector of an end-to-end learned function bases, i.e., a dictionary. Using this equivalence, we further establish that by tuning only the regularization factor, such parallel NN achieves an estimation error arbitrarily close to the minimax rates for both the Besov and BV classes. Notably, it gets exponentially closer to minimax optimal as the NN gets deeper. Our research sheds new lights on why depth matters and how NNs are more powerful than kernel methods.
Paper Structure (34 sections, 18 theorems, 121 equations, 7 figures, 2 tables)

This paper contains 34 sections, 18 theorems, 121 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Notation and Problem Setup.
  4. Besov Spaces and Bound Variation Space
  5. Main Results: Parallel ReLU DNNs
  6. Proof Overview
  7. Equivalence to $\ell_p$ Sparse Regression
  8. Estimation Error Analysis
  9. Approximation Error Analysis
  10. Experiment
  11. Conclusion and Discussion
  12. Other related works
  13. Two-layer Neural Network with Truncated Power Activation Functions
  14. Introduction To Common Function Classes
  15. Besov Class
  16. ...and 19 more sections

Key Result

Theorem 1

For any fixed $\alpha - d/p > 1, q \geq 1, L \geq 3$, define $m = \lceil \alpha - 1 \rceil$. For any $f_0 \in B^{\alpha}_{p, q}$, given an $L$-layer parallel neural network satisfying Under the assumption as in lemma:cons2regu, with proper choice of the parameter of regularizaton $\lambda$ that depends on ${\mathcal{D}}, \alpha, d, L$, the solution $\hat{f}$ parameterized by eq:l2 satisfies wher

Figures (7)

  • Figure 1: Illustration of a function with heterogeneous smoothness and the problem of locally adaptive nonparametric regression.
  • Figure 2: Parallel neural network and the equivalent sparse regression model we discovered.
  • Figure 3: Numerical experiment results of the Doppler function (a-c,h), and "vary" function (d-f,g). All the "active" subnetworks are plotted in (c)(f). The horizontal axis in (b) is not linear.
  • Figure 4: The relationship between degree of freedom and the scaling factor of the regularizer $\lambda$. The solid line shows the result after denoising. (a)(b)in a parallel NN. (c)(d) In trend filtering. (a)(c): the "vary" function. (b)(d) the doppler function.
  • Figure 5: More experiments results of Doppler function.
  • ...and 2 more figures

Theorems & Definitions (44)

  • Theorem 1
  • Corollary 2
  • Corollary 3
  • Proposition 4
  • Theorem 5
  • Lemma 6
  • Proposition 7
  • Theorem 8
  • Theorem 9
  • proof
  • ...and 34 more