Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signal-to-noise ratio aware minimaxity and higher-order asymptotics

Yilin Guo, Haolei Weng, Arian Maleki

TL;DR

Theoretical findings obtained from this refined minimax framework provide new insights and practical guidance for the estimation of sparse signals and discover minimax estimators.

Abstract

Since its development, the minimax framework has been one of the corner stones of theoretical statistics, and has contributed to the popularity of many well-known estimators, such as the regularized M-estimators for high-dimensional problems. In this paper, we will first show through the example of sparse Gaussian sequence model, that the theoretical results under the classical minimax framework are insufficient for explaining empirical observations. In particular, both hard and soft thresholding estimators are (asymptotically) minimax, however, in practice they often exhibit sub-optimal performances at various signal-to-noise ratio (SNR) levels. The first contribution of this paper is to demonstrate that this issue can be resolved if the signal-to-noise ratio is taken into account in the construction of the parameter space. We call the resulting minimax framework the signal-to-noise ratio aware minimaxity. The second contribution of this paper is to showcase how one can use higher-order asymptotics to obtain accurate approximations of the SNR-aware minimax risk and discover minimax estimators. The theoretical findings obtained from this refined minimax framework provide new insights and practical guidance for the estimation of sparse signals.

Signal-to-noise ratio aware minimaxity and higher-order asymptotics

TL;DR

Theoretical findings obtained from this refined minimax framework provide new insights and practical guidance for the estimation of sparse signals and discover minimax estimators.

Abstract

Since its development, the minimax framework has been one of the corner stones of theoretical statistics, and has contributed to the popularity of many well-known estimators, such as the regularized M-estimators for high-dimensional problems. In this paper, we will first show through the example of sparse Gaussian sequence model, that the theoretical results under the classical minimax framework are insufficient for explaining empirical observations. In particular, both hard and soft thresholding estimators are (asymptotically) minimax, however, in practice they often exhibit sub-optimal performances at various signal-to-noise ratio (SNR) levels. The first contribution of this paper is to demonstrate that this issue can be resolved if the signal-to-noise ratio is taken into account in the construction of the parameter space. We call the resulting minimax framework the signal-to-noise ratio aware minimaxity. The second contribution of this paper is to showcase how one can use higher-order asymptotics to obtain accurate approximations of the SNR-aware minimax risk and discover minimax estimators. The theoretical findings obtained from this refined minimax framework provide new insights and practical guidance for the estimation of sparse signals.
Paper Structure (45 sections, 37 theorems, 230 equations, 5 figures)

This paper contains 45 sections, 37 theorems, 230 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Classical minimaxity and its limitations in sparse Gaussian sequence model
  4. Our contributions and paper structure
  5. SNR-aware minimaxity
  6. SNR-aware minimax framework
  7. First order analysis of SNR-aware minimaxity and its drawbacks
  8. Second order analysis of SNR-aware minimaxity
  9. Results in Regime (i@)
  10. Results in Regime (ii@)
  11. Results in Regime (iii@)
  12. Numerical experiments
  13. Discussions
  14. Summary
  15. Related works
  16. ...and 30 more sections

Key Result

Theorem 1

Assume model model::gaussian_sequence and parameter space model::sparse-parameter-space with $k_n/n \rightarrow 0$ as $n\rightarrow \infty$. Then the minimax risk, defined in eq::sparse-minimax, satisfies Moreover, both the soft and hard thresholding estimators with tuning $\lambda_n=\sigma_{n}\sqrt{2\log (n/k)}$ are asymptotically minimax, i.e., for $\hat{\theta}=\hat{\eta}_S(y,\lambda_n)$ or $\

Figures (5)

  • Figure 1: Mean squared error comparison at different noise levels. Data is generated according to \ref{['model::gaussian_sequence']} with $k_n= \lfloor n^{2/3}\rfloor$ and $\theta$ having $k_n$ components equal to $1.5$. "linear" denotes the simple linear estimator $\frac{1}{1+\lambda}y$. All the three estimators are optimally tuned. MSE is averaged over 20 repetitions along with standard error. Other details of the simulation can be found in Section \ref{['simulations']}.
  • Figure 2: Mean squared error comparison at different noise levels. On each graph, the y-axis is the scaled MSE, and the x-axis is the noise standard deviation $\sigma_n$. (to be continued in Fig. \ref{['fig:MSEvsSigma2']})
  • Figure 3: (continued from Fig. \ref{['fig:MSEvsSigma1']})
  • Figure 4: Mean squared error comparison at different SNR levels. On each graph, the $y$-axis is the scaled MSE, and the $x$-axis is the SNR $\mu_n$. (to be continued in Fig. \ref{['fig:MSEvsMu2']})
  • Figure 5: (continued from Fig. \ref{['fig:MSEvsMu1']})

Theorems & Definitions (56)

  • Theorem 1: donoho1994minimaxdonoho1992maximumjohnstone19
  • Theorem 2
  • Theorem 3
  • Proposition 1
  • Proposition 2
  • Theorem 4
  • Remark 1
  • Theorem 5
  • Proposition 3
  • Proposition 4
  • ...and 46 more