Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inference for Deep Neural Network Estimators in Generalized Nonparametric Models

Xuran Meng, Yi Li

TL;DR

The paper develops a rigorous framework for inferring subject-specific means estimated by deep neural networks under generalized nonparametric regression models, explicitly allowing covariate-dependent, heteroskedastic residuals. It introduces the Ensemble Subsampling Method (ESM), which leverages U-statistics and Hoeffding decomposition to obtain model-free variance estimates and valid confidence intervals for DNN-based mean estimators. Theoretical results provide convergence guarantees for the DNN estimator under GNRMs, asymptotic normality of the ensemble predictor, and consistency of the variance estimator, with empirical validation through simulations and application to the eICU dataset showing practical utility for personalized clinical decision making. The framework enables uncertainty quantification for a broad class of functionals of the conditional mean, with potential extensions to causal inference and variance-based targets, thereby broadening the impact of DNN-based inference in nonparametric contexts.

Abstract

While deep neural networks (DNNs) are used for prediction, inference on DNN-estimated subject-specific means for categorical or exponential family outcomes remains underexplored. We address this by proposing a DNN estimator under generalized nonparametric regression models (GNRMs) and developing a rigorous inference framework. Unlike existing approaches that assume independence between estimation errors and inputs to establish the error bound, a condition often violated in GNRMs, we allow for dependence and our theoretical analysis demonstrates the feasibility of drawing inference under GNRMs. To implement inference, we consider an Ensemble Subsampling Method (ESM) that leverages U-statistics and the Hoeffding decomposition to construct reliable confidence intervals for DNN estimates. We show that, under GNRM settings, ESM enables model-free variance estimation and accounts for heterogeneity among individuals in the population. Through simulations under nonparametric logistic, Poisson, and binomial regression models, we demonstrate the effectiveness and efficiency of our method. We further apply the method to the electronic Intensive Care Unit (eICU) dataset, a large scale collection of anonymized health records from ICU patients, to predict ICU readmission risk and offer patient-centric insights for clinical decision making.

Inference for Deep Neural Network Estimators in Generalized Nonparametric Models

TL;DR

The paper develops a rigorous framework for inferring subject-specific means estimated by deep neural networks under generalized nonparametric regression models, explicitly allowing covariate-dependent, heteroskedastic residuals. It introduces the Ensemble Subsampling Method (ESM), which leverages U-statistics and Hoeffding decomposition to obtain model-free variance estimates and valid confidence intervals for DNN-based mean estimators. Theoretical results provide convergence guarantees for the DNN estimator under GNRMs, asymptotic normality of the ensemble predictor, and consistency of the variance estimator, with empirical validation through simulations and application to the eICU dataset showing practical utility for personalized clinical decision making. The framework enables uncertainty quantification for a broad class of functionals of the conditional mean, with potential extensions to causal inference and variance-based targets, thereby broadening the impact of DNN-based inference in nonparametric contexts.

Abstract

While deep neural networks (DNNs) are used for prediction, inference on DNN-estimated subject-specific means for categorical or exponential family outcomes remains underexplored. We address this by proposing a DNN estimator under generalized nonparametric regression models (GNRMs) and developing a rigorous inference framework. Unlike existing approaches that assume independence between estimation errors and inputs to establish the error bound, a condition often violated in GNRMs, we allow for dependence and our theoretical analysis demonstrates the feasibility of drawing inference under GNRMs. To implement inference, we consider an Ensemble Subsampling Method (ESM) that leverages U-statistics and the Hoeffding decomposition to construct reliable confidence intervals for DNN estimates. We show that, under GNRM settings, ESM enables model-free variance estimation and accounts for heterogeneity among individuals in the population. Through simulations under nonparametric logistic, Poisson, and binomial regression models, we demonstrate the effectiveness and efficiency of our method. We further apply the method to the electronic Intensive Care Unit (eICU) dataset, a large scale collection of anonymized health records from ICU patients, to predict ICU readmission risk and offer patient-centric insights for clinical decision making.

Paper Structure

This paper contains 36 sections, 12 theorems, 163 equations, 14 figures, 20 tables.

Table of Contents

  1. Introduction
  2. The Preamble
  3. Deep Neural Networks
  4. Generalized Nonparametric Regression
  5. Empirical Risk Minimization and Function Smoothness
  6. Ensemble Subsampling Method (ESM)
  7. Theoretical Results
  8. Numerical Experiments
  9. Analysis of eICU data
  10. Discussion
  11. Additional Numerical Results
  12. Binomial Regression
  13. Comparison to Random Forests
  14. Additional Experiments on Kernel Regressions
  15. Additional Experiments on Dimensionality and Nonlinearity
  16. ...and 21 more sections

Key Result

Theorem 3.5

Suppose that $f_0\in \mathcal{G}(q,\mathbf{d},\mathbf{t},{\bm{\gamma}},K)$ and $\widehat{f}_n\in\mathcal{F}(L,\mathbf{p},s,F)$. If Assumptions assump:1-assump:network hold, then it holds that for some constants $c',C'>0$. Here, $\Delta_{n}(\widehat{f}_n)$ is defined in eq:obtained_estimator.

Figures (14)

  • Figure 1: Overview of the Ensemble Subsampling Method (ESM).
  • Figure 2: Estimation and inference in simulation samples: Logistic Model with $n=700$, $r=n^{0.9}$ and $B=1400$. Figure \ref{['fig_simu:part1CI']} shows the average estimated $\mathbb{E}(y|\mathbf{x})$ with variability across 300 repetitions (gray band) over test points. Figure \ref{['fig_simu:part1SD']} compares corrected and uncorrected standard errors and their variability (gray and blue bands, respectively) to the empirical standard deviations of estimates across all test samples.
  • Figure 3: Estimation and inference in simulation samples: Logistic Model with $n=400$ and $r=n^{0.8}$ under a deeper network. Figure \ref{['fig_simu:part3CI']} displays the average estimated $\mathbb{E}(y|\mathbf{x})$ and its deviation (gray band) across 300 runs at various test points. Figure \ref{['fig_simu:part3SD']} compares corrected and uncorrected SEs and their deviations (gray and blue bands) across 300 runs with empirical SDs across test samples.
  • Figure 4: Evaluation of the nonparametric logistic model estimator. Figure \ref{['fig_real:rocNNlog']} and \ref{['fig_real:rocRFlog']} show the ROC curve with both AUC of 0.84. Figure \ref{['fig_real:calibrationLogisticNN']} and \ref{['fig_real:calibrationLogisticRF']} display estimated subject-level probabilities of ICU readmission and confidence intervals, illustrating heteroskedasticity across individuals.
  • Figure 5: Evaluation of the nonparametric logistic model estimator. Figure \ref{['fig_real:rocNNpoi']} and \ref{['fig_real:rocRFpoi']} show the ROC curve with both AUC of 0.72. Figure \ref{['fig_real:calibrationPoissonNN']} and \ref{['fig_real:calibrationPoissonRF']} display estimated subject-level probabilities of ICU readmission and confidence intervals, illustrating heteroskedasticity across individuals.
  • ...and 9 more figures

Theorems & Definitions (12)

  • Theorem 3.5
  • Corollary 3.6
  • Theorem 3.7
  • Theorem 3.8
  • Proposition C.1
  • Lemma C.2
  • Lemma C.3
  • Lemma C.4
  • Lemma D.1
  • Lemma D.2
  • ...and 2 more