Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Observable adjustments in single-index models for regularized M-estimators

Pierre C Bellec

TL;DR

This work develops data-driven observable adjustments that transform the high-dimensional inference for regularized M-estimators in single-index models into proximal representations governed by observable quantities. It provides confidence intervals for individual index components and estimators of the index correlation without requiring knowledge of the latent link or index distribution, and it does so for both strongly convex penalties and unregularized M-estimation. The approach yields proximal mappings for the estimator and predictions, along with accurate data-driven estimates of correlation terms, enabling practical inference in high dimensions. The theory is supported by simulations across square loss, logistic loss, and 1-bit compressed sensing, illustrating robust, data-driven inference in diverse single-index settings and informing hyperparameter tuning through correlation-focused criteria.

Abstract

We consider observations $(X,y)$ from single index models with unknown link function, Gaussian covariates and a regularized M-estimator $\hatβ$ constructed from convex loss function and regularizer. In the regime where sample size $n$ and dimension $p$ are both increasing such that $p/n$ has a finite limit, the behavior of the empirical distribution of $\hatβ$ and the predicted values $X\hatβ$ has been previously characterized in a number of models: The empirical distributions are known to converge to proximal operators of the loss and penalty in a related Gaussian sequence model, which captures the interplay between ratio $p/n$, loss, regularization and the data generating process. This connection between$(\hatβ,X\hatβ)$ and the corresponding proximal operators require solving fixed-point equations that typically involve unobservable quantities such as the prior distribution on the index or the link function. This paper develops a different theory to describe the empirical distribution of $\hatβ$ and $X\hatβ$: Approximations of $(\hatβ,X\hatβ)$ in terms of proximal operators are provided that only involve observable adjustments. These proposed observable adjustments are data-driven, e.g., do not require prior knowledge of the index or the link function. These new adjustments yield confidence intervals for individual components of the index, as well as estimators of the correlation of $\hatβ$ with the index. The interplay between loss, regularization and the model is thus captured in a data-driven manner, without solving the fixed-point equations studied in previous works. The results apply to both strongly convex regularizers and unregularized M-estimation. Simulations are provided for the square and logistic loss in single index models including logistic regression and 1-bit compressed sensing with 20\% corrupted bits.

Observable adjustments in single-index models for regularized M-estimators

TL;DR

This work develops data-driven observable adjustments that transform the high-dimensional inference for regularized M-estimators in single-index models into proximal representations governed by observable quantities. It provides confidence intervals for individual index components and estimators of the index correlation without requiring knowledge of the latent link or index distribution, and it does so for both strongly convex penalties and unregularized M-estimation. The approach yields proximal mappings for the estimator and predictions, along with accurate data-driven estimates of correlation terms, enabling practical inference in high dimensions. The theory is supported by simulations across square loss, logistic loss, and 1-bit compressed sensing, illustrating robust, data-driven inference in diverse single-index settings and informing hyperparameter tuning through correlation-focused criteria.

Abstract

We consider observations from single index models with unknown link function, Gaussian covariates and a regularized M-estimator constructed from convex loss function and regularizer. In the regime where sample size and dimension are both increasing such that has a finite limit, the behavior of the empirical distribution of and the predicted values has been previously characterized in a number of models: The empirical distributions are known to converge to proximal operators of the loss and penalty in a related Gaussian sequence model, which captures the interplay between ratio , loss, regularization and the data generating process. This connection between and the corresponding proximal operators require solving fixed-point equations that typically involve unobservable quantities such as the prior distribution on the index or the link function. This paper develops a different theory to describe the empirical distribution of and : Approximations of in terms of proximal operators are provided that only involve observable adjustments. These proposed observable adjustments are data-driven, e.g., do not require prior knowledge of the index or the link function. These new adjustments yield confidence intervals for individual components of the index, as well as estimators of the correlation of with the index. The interplay between loss, regularization and the model is thus captured in a data-driven manner, without solving the fixed-point equations studied in previous works. The results apply to both strongly convex regularizers and unregularized M-estimation. Simulations are provided for the square and logistic loss in single index models including logistic regression and 1-bit compressed sensing with 20\% corrupted bits.
Paper Structure (39 sections, 30 theorems, 218 equations, 1 figure, 3 tables)

This paper contains 39 sections, 30 theorems, 218 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Prior works in asymptotic behavior of M-estimators
  3. Contributions
  4. Organization
  5. Notation
  6. Assumptions
  7. Derivatives and observable adjustments
  8. Definition of the observable adjustments
  9. Main results under strong convexity
  10. Confidence intervals for individual components of the index
  11. Proximal mapping representation for $\hat{\beta}_j$
  12. Proximal mapping representation for predicted values
  13. Correlation estimation
  14. Sign disambiguation: estimating $\text{sign}(t_*)$ and $\text{sign}(a_*)$
  15. Main results for unregularized $M$-estimation
  16. ...and 24 more sections

Key Result

Proposition 3.0

Assume that $\ell_{y_0}$ is convex with $\ell_{y_0}'$ 1-Lipschitz for all $y_0\in\mathcal{Y}$ and that eqStrongConvex holds for some some $\tau > 0$ and positive definite $\bm{\Sigma}\in\mathbb{R}^{p\times p}$. Then for any fixed $\bm{y}\in \mathcal{Y}^n$ the function $\bm{\hat{\beta}}(\bm y, \cdot) where $\bm{D}\stackrel{\text{\tiny def}}{=}\mathop{\mathrm{diag}}\nolimits(\ell_{\bm{y}}"(\bm{X}\bm

Figures (1)

  • Figure 1: Left: Correlation $a_*=\bm w^T\bm\Sigma\bm\hat{\beta}$ and its estimate $\sqrt{\hat{a}^2}$ for $\hat{a}^2$ in \ref{['hat_a_L1']} for $n=1000$, in the binomial model $y_i|\bm x_i\sim\text{Binomial}(q,\rho'(\bm x_i^T\bm\beta^*))$ and the L1-penalized M-estimator \ref{['hbeta-L1']}. The precise simulation setting is described in \ref{['sec:L1']}. The x-axis represents the tuning parameter $\lambda$ in \ref{['hbeta-L1']}.

Theorems & Definitions (47)

  • Proposition 3.0
  • Theorem 4.1
  • Corollary 4.1
  • proof
  • Theorem 4.2
  • Theorem 4.3
  • Proposition 4.4
  • Lemma 5.0
  • Theorem 5.1
  • Corollary 5.2
  • ...and 37 more