Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semiparametric M-estimation with overparameterized neural networks

Shunxing Yan, Ziyuan Chen, Fang Yao

TL;DR

This work develops a foundational framework for semiparametric M-estimation using overparameterized neural networks to enable interpretable inference for finite-dimensional parameters while exploiting DNN flexibility. By linking DNN training to an RKHS surrogate via the neural tangent kernel and analyzing gradient-flow dynamics, the authors prove nonparametric convergence for the network component and $\sqrt{n}$-consistency with asymptotic normality for the parametric component under a Huberized margin condition that accommodates unbounded nonparametric parts. The results hold without requiring bounded network outputs or true functions to lie in the RKHS, and they establish efficiency in likelihood-based settings when the least favorable submodel is present. The theoretical findings are complemented by regression and classification examples and numerical experiments demonstrating favorable finite-sample performance and valid inferential coverage, supporting the practical applicability of semiparametric neural M-estimation with overparameterized networks.

Abstract

We focus on semiparametric regression that has played a central role in statistics, and exploit the powerful learning ability of deep neural networks (DNNs) while enabling statistical inference on parameters of interest that offers interpretability. Despite the success of classical semiparametric method/theory, establishing the $\sqrt{n}$-consistency and asymptotic normality of the finite-dimensional parameter estimator in this context remains challenging, mainly due to nonlinearity and potential tangent space degeneration in DNNs. In this work, we introduce a foundational framework for semiparametric $M$-estimation, leveraging the approximation ability of overparameterized neural networks that circumvent tangent degeneration and align better with training practice nowadays. The optimization properties of general loss functions are analyzed, and the global convergence is guaranteed. Instead of studying the ``ideal'' solution to minimization of an objective function in most literature, we analyze the statistical properties of algorithmic estimators, and establish nonparametric convergence and parametric asymptotic normality for a broad class of loss functions. These results hold without assuming the boundedness of the network output and even when the true function lies outside the specified function space. To illustrate the applicability of the framework, we also provide examples from regression and classification, and the numerical experiments provide empirical support to the theoretical findings.

Semiparametric M-estimation with overparameterized neural networks

TL;DR

This work develops a foundational framework for semiparametric M-estimation using overparameterized neural networks to enable interpretable inference for finite-dimensional parameters while exploiting DNN flexibility. By linking DNN training to an RKHS surrogate via the neural tangent kernel and analyzing gradient-flow dynamics, the authors prove nonparametric convergence for the network component and -consistency with asymptotic normality for the parametric component under a Huberized margin condition that accommodates unbounded nonparametric parts. The results hold without requiring bounded network outputs or true functions to lie in the RKHS, and they establish efficiency in likelihood-based settings when the least favorable submodel is present. The theoretical findings are complemented by regression and classification examples and numerical experiments demonstrating favorable finite-sample performance and valid inferential coverage, supporting the practical applicability of semiparametric neural M-estimation with overparameterized networks.

Abstract

We focus on semiparametric regression that has played a central role in statistics, and exploit the powerful learning ability of deep neural networks (DNNs) while enabling statistical inference on parameters of interest that offers interpretability. Despite the success of classical semiparametric method/theory, establishing the -consistency and asymptotic normality of the finite-dimensional parameter estimator in this context remains challenging, mainly due to nonlinearity and potential tangent space degeneration in DNNs. In this work, we introduce a foundational framework for semiparametric -estimation, leveraging the approximation ability of overparameterized neural networks that circumvent tangent degeneration and align better with training practice nowadays. The optimization properties of general loss functions are analyzed, and the global convergence is guaranteed. Instead of studying the ``ideal'' solution to minimization of an objective function in most literature, we analyze the statistical properties of algorithmic estimators, and establish nonparametric convergence and parametric asymptotic normality for a broad class of loss functions. These results hold without assuming the boundedness of the network output and even when the true function lies outside the specified function space. To illustrate the applicability of the framework, we also provide examples from regression and classification, and the numerical experiments provide empirical support to the theoretical findings.
Paper Structure (13 sections, 7 theorems, 52 equations, 4 tables)

This paper contains 13 sections, 7 theorems, 52 equations, 4 tables.

Table of Contents

  1. Introduction
  2. Literature review
  3. Challenges and main contributions
  4. Notations and organization
  5. Semiparametric M-estimation with neural networks
  6. Overparameterized neural network and neural tangent kernel
  7. Semiparametric neural M-estimation
  8. Gradient flow optimization and convergence
  9. Statistical theory of algorithmic neural M-estimation
  10. Examples
  11. Regression
  12. Classification
  13. Numerical studies

Key Result

Proposition 2.1

Under the random initialization mechanism proposed for $W_i$'s and $b_i$'s, the neural tangent kernel has the explicit expression as where $\tilde{u} := ({\boldsymbol{x}}^{T}{\boldsymbol{x}}' + 1)/\sqrt{ (\|\boldsymbol{x}\|_2^2+1) (\|\boldsymbol{x} ' \|_2^2+1) }$, $\kappa_0(t)$ and $\kappa_1(t)$ are arc-cosine kernels of degree $0$ and $1$, i.e. and $h^{(r)}, r \geq 1$ denote the $r$-times compo

Theorems & Definitions (9)

  • Proposition 2.1
  • Proposition 2.2
  • Remark 2.1
  • Proposition 2.3
  • Remark 2.2
  • Theorem 2.1
  • Theorem 3.1
  • Theorem 4.1
  • Theorem 4.2