Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimality and computational barriers in variable selection under dependence

Ming Gao, Bryon Aragam

TL;DR

This work analyzes variable selection in Gaussian linear models with general design covariance, establishing that best subset selection (BSS) is minimax optimal for exact support recovery and deriving the precise sample complexity dependence on dimension, sparsity, signal strength, and design dependence. It extends optimality to unknown sparsity using an information-criterion-type estimator and uncovers a fundamental statistical-computational trade-off governed by restricted eigenvalues, showing that efficient algorithms cannot match BSS under general dependence. A polynomial-time lower bound demonstrates a computability gap, suggesting that, without strong covariance conditions, efficient methods cannot achieve the same statistical efficiency as BSS. The results highlight the pivotal role of restricted eigenvalues and contribute a finite-sample information-criterion analysis of variable selection with general covariance.

Abstract

We study the optimal sample complexity of variable selection in linear regression under general design covariance, and show that subset selection is optimal while under standard complexity assumptions, efficient algorithms for this problem do not exist. Specifically, we analyze the variable selection problem and provide the optimal sample complexity with exact dependence on the problem parameters for both known and unknown sparsity settings. Moreover, we establish a sample complexity lower bound for any efficient estimator, highlighting a gap between the statistical efficiency achievable by combinatorial algorithms (such as subset selection) compared to efficient algorithms (such as those based on convex programming). The proofs rely on a finite-sample analysis of an information criterion estimator, which may be of independent interest. Our results emphasize the optimal position of subset selection, the critical role played by restricted eigenvalues, and characterize the statistical-computational trade-off in high-dimensional variable selection.

Optimality and computational barriers in variable selection under dependence

TL;DR

This work analyzes variable selection in Gaussian linear models with general design covariance, establishing that best subset selection (BSS) is minimax optimal for exact support recovery and deriving the precise sample complexity dependence on dimension, sparsity, signal strength, and design dependence. It extends optimality to unknown sparsity using an information-criterion-type estimator and uncovers a fundamental statistical-computational trade-off governed by restricted eigenvalues, showing that efficient algorithms cannot match BSS under general dependence. A polynomial-time lower bound demonstrates a computability gap, suggesting that, without strong covariance conditions, efficient methods cannot achieve the same statistical efficiency as BSS. The results highlight the pivotal role of restricted eigenvalues and contribute a finite-sample information-criterion analysis of variable selection with general covariance.

Abstract

We study the optimal sample complexity of variable selection in linear regression under general design covariance, and show that subset selection is optimal while under standard complexity assumptions, efficient algorithms for this problem do not exist. Specifically, we analyze the variable selection problem and provide the optimal sample complexity with exact dependence on the problem parameters for both known and unknown sparsity settings. Moreover, we establish a sample complexity lower bound for any efficient estimator, highlighting a gap between the statistical efficiency achievable by combinatorial algorithms (such as subset selection) compared to efficient algorithms (such as those based on convex programming). The proofs rely on a finite-sample analysis of an information criterion estimator, which may be of independent interest. Our results emphasize the optimal position of subset selection, the critical role played by restricted eigenvalues, and characterize the statistical-computational trade-off in high-dimensional variable selection.

Paper Structure

This paper contains 15 sections, 17 theorems, 103 equations.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Notation
  5. Preliminaries
  6. Optimality with known sparsity
  7. Optimality with unknown sparsity
  8. Polynomial-efficient sample complexity lower bound
  9. Conclusion
  10. Proof of Lemma \ref{['lem:opt:bss:ub:generic']}
  11. Proof of Theorem \ref{['thm:opt:bss:lb1']}
  12. Proof of Theorem \ref{['thm:opt:bss:ub:unknown']}
  13. Proof of Theorem \ref{['thm:opt:bss:lb1:unknown']}
  14. Proof of Lemma \ref{['thm:poly:main']}
  15. Auxiliary lemmas

Key Result

Lemma 3.1

Assuming $s\le d/2$, for any $(\beta,\Sigma,\sigma^2) \in \mathcal{M}$, let $S_{*}=\mathop{\mathrm{supp}}\nolimits(\beta)$, given $n$ i.i.d. samples from $P_{\beta,\Sigma,\sigma^2}$, if the sample size then $\mathbb{P}_{\beta,\Sigma,\sigma^2}(\widehat{S}^{\text{BSS}{}}=S_{*})\ge 1-\delta$.

Theorems & Definitions (34)

  • Remark 2.1
  • Definition 1
  • Lemma 3.1
  • Theorem 3.2: wainwright2009information, Theorem 1
  • proof
  • Theorem 3.3
  • Theorem 3.4: wang2010informationl, Theorem 1
  • Theorem 4.1
  • Remark 4.1
  • Theorem 4.2
  • ...and 24 more