Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signal-to-noise ratio aware minimax analysis of sparse linear regression

Shubhangi Ghosh, Yilin Guo, Haolei Weng, Arian Maleki

TL;DR

This paper addresses minimax risk in high-dimensional sparse linear regression under Gaussian design by introducing an SNR-aware framework that constrains signal strength with $\Theta(k,\tau)$. By analyzing risk as a function of the SNR parameter $\mu=\tau/\sigma$, it identifies three distinct SNR regimes and provides both first- and second-order asymptotics: Regimes I/II yield $R(\Theta(k,\tau),\sigma)=k\tau^2(1+o(1))$, while Regime III follows $R(\Theta(k,\tau),\sigma)=2\sigma^2 k\log(p/k)(1+o(1))$; the second-order results show ridge is optimal in low SNR (Regime I), while elastic-net becomes near-minimax optimal in moderate SNR (Regime II). The simulations corroborate the theory, showing ridge, elastic-net, and Lasso dominate in low, medium, and high SNR, respectively, explaining empirical observations that classic rate-optimal minimax results miss. Overall, the work provides a refined view of estimator performance across SNR levels and offers guidance for method selection in noisy, high-dimensional settings.

Abstract

We consider parameter estimation under sparse linear regression -- an extensively studied problem in high-dimensional statistics and compressed sensing. While the minimax framework has been one of the most fundamental approaches for studying statistical optimality in this problem, we identify two important issues that the existing minimax analyses face: (i) The signal-to-noise ratio appears to have no effect on the minimax optimality, while it shows a major impact in numerical simulations. (ii) Estimators such as best subset selection and Lasso are shown to be minimax optimal, yet they exhibit significantly different performances in simulations. In this paper, we tackle the two issues by employing a minimax framework that accounts for variations in the signal-to-noise ratio (SNR), termed the SNR-aware minimax framework. We adopt a delicate higher-order asymptotic analysis technique to obtain the SNR-aware minimax risk. Our theoretical findings determine three distinct SNR regimes: low-SNR, medium-SNR, and high-SNR, wherein minimax optimal estimators exhibit markedly different behaviors. The new theory not only offers much better elaborations for empirical results, but also brings new insights to the estimation of sparse signals in noisy data.

Signal-to-noise ratio aware minimax analysis of sparse linear regression

TL;DR

This paper addresses minimax risk in high-dimensional sparse linear regression under Gaussian design by introducing an SNR-aware framework that constrains signal strength with . By analyzing risk as a function of the SNR parameter , it identifies three distinct SNR regimes and provides both first- and second-order asymptotics: Regimes I/II yield , while Regime III follows ; the second-order results show ridge is optimal in low SNR (Regime I), while elastic-net becomes near-minimax optimal in moderate SNR (Regime II). The simulations corroborate the theory, showing ridge, elastic-net, and Lasso dominate in low, medium, and high SNR, respectively, explaining empirical observations that classic rate-optimal minimax results miss. Overall, the work provides a refined view of estimator performance across SNR levels and offers guidance for method selection in noisy, high-dimensional settings.

Abstract

We consider parameter estimation under sparse linear regression -- an extensively studied problem in high-dimensional statistics and compressed sensing. While the minimax framework has been one of the most fundamental approaches for studying statistical optimality in this problem, we identify two important issues that the existing minimax analyses face: (i) The signal-to-noise ratio appears to have no effect on the minimax optimality, while it shows a major impact in numerical simulations. (ii) Estimators such as best subset selection and Lasso are shown to be minimax optimal, yet they exhibit significantly different performances in simulations. In this paper, we tackle the two issues by employing a minimax framework that accounts for variations in the signal-to-noise ratio (SNR), termed the SNR-aware minimax framework. We adopt a delicate higher-order asymptotic analysis technique to obtain the SNR-aware minimax risk. Our theoretical findings determine three distinct SNR regimes: low-SNR, medium-SNR, and high-SNR, wherein minimax optimal estimators exhibit markedly different behaviors. The new theory not only offers much better elaborations for empirical results, but also brings new insights to the estimation of sparse signals in noisy data.
Paper Structure (30 sections, 37 theorems, 243 equations, 3 figures)

This paper contains 30 sections, 37 theorems, 243 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Limitations of the minimax framework
  3. Our approach: SNR-aware minimaxity
  4. Notations
  5. SNR-aware minimaxity
  6. Definition of the SNR-aware framework
  7. First-order asymptotic approximation
  8. Second-order asymptotic approximation
  9. Second-order approximation in Regime i@
  10. Second-order approximation in Regime ii@
  11. Simulations
  12. Discussions
  13. Related works
  14. Conclusion
  15. Preliminaries
  16. ...and 15 more sections

Key Result

Theorem 1

Assume model model::gaussian-model and parameter space param::sparse. As $k/p \rightarrow 0$ and $(k \log p)/n \rightarrow 0$, the minimax risk defined in eq::sparse-minimax satisfiesThe high-probability version of this result appeared earlier in su2016slope. This minimax risk is (asymptotically) achieved by switching between the best subset selection and Lasso based on a certain switching rule.

Figures (3)

  • Figure 1: Mean-squared error comparison at different SNR values. Data is generated according to \ref{['model::gaussian-model']}. MSE is defined as the average of $\|\hat{\beta}-\beta\|_2^2/\|\beta\|_2^2$ over 50 experiments. We set $\beta$ as, for a randomly sampled index set $S$ of cardinality $k$, $\beta_i = \tau \,\forall i\in S; \, \beta_i=0$ otherwise. SNR $\coloneqq \tau/\sigma$. More simulation results are provided in Section \ref{['simulation:sec']}. To facilitate a clearer comparison between ridge, LASSO, and subset selection, we limit the MSE range to $[0, 1]$ in the left panel. However, the right panel displays the MSE of best subset selection across the full range of $1/{\rm SNR}$.
  • Figure 2: Mean-squared error comparison at different SNR values.
  • Figure 3: Mean-squared error comparison at different SNR values.

Theorems & Definitions (70)

  • Theorem 1: guo2024minimaxlr
  • Theorem 2
  • Remark 1
  • Remark 2
  • Proposition 1
  • Theorem 3
  • Remark 3
  • Remark 4
  • Theorem 4
  • Remark 5
  • ...and 60 more