Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A general framework for deep learning

William Kengne, Modou Wade

TL;DR

This work presents a unified framework for deep learning in nonparametric regression and classification when data exhibit dependence captured by a generalized Bernstein-type inequality. It introduces two estimators, NPDNN and SPDNN, and derives excess-risk bounds for Hölder and composition Hölder function classes, showing minimax-optimal rates up to logarithmic factors across i.i.d. and several mixing processes. The paper also provides an oracle inequality for SPDNN, with practical guidance on tuning parameters to balance approximation and sparsity, and demonstrates applicability to nonparametric autoregression with exogenous covariates. Across theoretical results and applications, the framework accommodates a broad spectrum of dependence structures, offering rates that align with known minimax limits in several classical settings. The work advances understanding of how dependent data affect deep-learning convergence and highlights sparsity-penalized DNNs as robust, near-optimal predictors in complex stochastic environments.

Abstract

This paper develops a general approach for deep learning for a setting that includes nonparametric regression and classification. We perform a framework from data that fulfills a generalized Bernstein-type inequality, including independent, $φ$-mixing, strongly mixing and $\mathcal{C}$-mixing observations. Two estimators are proposed: a non-penalized deep neural network estimator (NPDNN) and a sparse-penalized deep neural network estimator (SPDNN). For each of these estimators, bounds of the expected excess risk on the class of Hölder smooth functions and composition Hölder functions are established. Applications to independent data, as well as to $φ$-mixing, strongly mixing, $\mathcal{C}$-mixing processes are considered. For each of these examples, the upper bounds of the expected excess risk of the proposed NPDNN and SPDNN predictors are derived. It is shown that both the NPDNN and SPDNN estimators are minimax optimal (up to a logarithmic factor) in many classical settings.

A general framework for deep learning

TL;DR

This work presents a unified framework for deep learning in nonparametric regression and classification when data exhibit dependence captured by a generalized Bernstein-type inequality. It introduces two estimators, NPDNN and SPDNN, and derives excess-risk bounds for Hölder and composition Hölder function classes, showing minimax-optimal rates up to logarithmic factors across i.i.d. and several mixing processes. The paper also provides an oracle inequality for SPDNN, with practical guidance on tuning parameters to balance approximation and sparsity, and demonstrates applicability to nonparametric autoregression with exogenous covariates. Across theoretical results and applications, the framework accommodates a broad spectrum of dependence structures, offering rates that align with known minimax limits in several classical settings. The work advances understanding of how dependent data affect deep-learning convergence and highlights sparsity-penalized DNNs as robust, near-optimal predictors in complex stochastic environments.

Abstract

This paper develops a general approach for deep learning for a setting that includes nonparametric regression and classification. We perform a framework from data that fulfills a generalized Bernstein-type inequality, including independent, -mixing, strongly mixing and -mixing observations. Two estimators are proposed: a non-penalized deep neural network estimator (NPDNN) and a sparse-penalized deep neural network estimator (SPDNN). For each of these estimators, bounds of the expected excess risk on the class of Hölder smooth functions and composition Hölder functions are established. Applications to independent data, as well as to -mixing, strongly mixing, -mixing processes are considered. For each of these examples, the upper bounds of the expected excess risk of the proposed NPDNN and SPDNN predictors are derived. It is shown that both the NPDNN and SPDNN estimators are minimax optimal (up to a logarithmic factor) in many classical settings.
Paper Structure (17 sections, 5 theorems, 65 equations)

This paper contains 17 sections, 5 theorems, 65 equations.

Table of Contents

  1. Introduction
  2. Notations and assumptions
  3. Some notations
  4. Some assumptions
  5. Excess risk bounds
  6. Excess risk bounds of the NPDNN estimator
  7. Excess risk bounds of the SPDNN estimator
  8. Applications and examples
  9. I.I.d. processes.
  10. $\phi$-mixing processes
  11. Strong mixing processes
  12. $\mathcal{C}$-mixing processes
  13. Nonparametric autoregression with exogenous covariate
  14. Proofs of the main results
  15. Proof of Theorem \ref{['thm2']}
  16. ...and 2 more sections

Key Result

Theorem 3.1

Assume (A0), (A1), (A2), (A4). Assume that (A3) is satisfied for some $\kappa \geq 1$, ${ \mathcal{K}} _0, \varepsilon_0>0$ and that $h^{*} \in \mathcal{C}^{s}(\mathcal{X}, \mathcal{K})$ for some $s, \mathcal{K} > 0$, where $h^{*}$ is defined in (best_pred_F). Set $L_n = \dfrac{s L_0}{\kappa s + d} for all $\nu>3$.

Theorems & Definitions (6)

  • Definition 2.1
  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Corollary 3.4
  • Lemma 6.1