Rethinking Hard Thresholding Pursuit: Full Adaptation and Sharp Estimation

TL;DR

The paper addresses high-dimensional sparse linear regression by rethinking hard thresholding methods through Full-Adaptive HTP (FAHTP), a tuning-free procedure that adapts to unknown sparsity and signal strength. It develops nonasymptotic error bounds for HTP under RIP, introduces an information-criterion–based adaptivity that yields minimax optimality with unknown $s^*$ and $\sigma$, and proves oracle-rate estimation and, under beta-min conditions, exact support recovery. When the beta-min condition is not met, FAHTP still delivers tighter-than-minimax error bounds via a general signal-analysis framework that partitions signals into strong and weak components. Numerical experiments, including a real data example, demonstrate the method’s robust performance, including convergence of minimum signal strength, effective adaptive tuning, and superiority over convex competitors in estimation and variable selection. Overall, FAHTP extends the favorable properties of HTP to practical, tuning-free high-dimensional estimation with strong theoretical guarantees and empirical validation.

Abstract

Hard Thresholding Pursuit (HTP) has aroused increasing attention for its robust theoretical guarantees and impressive numerical performance in non-convex optimization. In this paper, we introduce a novel tuning-free procedure, named Full-Adaptive HTP (FAHTP), that simultaneously adapts to both the unknown sparsity and signal strength of the underlying model. We provide an in-depth analysis of the iterative thresholding dynamics of FAHTP, offering refined theoretical insights. In specific, under the beta-min condition $\min_{i \in S^*}|{\boldsymbolβ}^*_i| \ge Cσ(\log p/n)^{1/2}$, we show that the FAHTP achieves oracle estimation rate $σ(s^*/n)^{1/2}$, highlighting its theoretical superiority over convex competitors such as LASSO and SLOPE, and recovers the true support set exactly. More importantly, even without the beta-min condition, our method achieves a tighter error bound than the classical minimax rate with high probability. The comprehensive numerical experiments substantiate our theoretical findings, underscoring the effectiveness and robustness of the proposed FAHTP.

Figures (6)

• Figure 1: The estimation error bound of HTP estimator with metric $\|\cdot\|_2^2$ for model size $s \in [s^*, 4s^*]$. Here $c$ is some positive constant.
• Figure 2: The solution path of HTP as iteration $t$ increases. The blue region indicates the iterations achieving the minimax rate, while the red region represents the iterations reaching the oracle rate.
• Figure 3: The minimum signal strength of estimator ${\boldsymbol{\beta}}^t$ for different model sizes under the conditions of Theorem \ref{['thm:lambda2']}.
• Figure 4: The phenomenon of the minimum signal strength of HTP estimators with varying model sizes. The horizontal black line represents the true minimal signal strength. (A) The convergence of the minimal strength of estimators within 20 iterations. (B) The minimum signal strength of estimators with different model sizes.
• Figure 5: Performance with increasing minimal signal strength.

Theorems & Definitions (22)

• Proposition 1
• Proposition 2
• Theorem 1
• Remark 1
• Corollary 1
• Theorem 2
• Remark 2
• Remark 3
• Theorem 3
• Remark 4