Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compressed Newton-direction-based Thresholding Methods for Sparse Optimization Problems

Nan Meng, Yun-Bin Zhao

TL;DR

Addresses sparse optimization with $\|x\|_0 \le k$ by introducing a compressed Newton direction that operates in a low-dimensional subspace and is embedded into the full space via diagonal regularization. Builds two algorithm families, CNHT/CNHTP (hard/ pursuit) and CNOT/CNOTP (optimal thresholding), integrated into a unified iterative framework $x^{(p+1)} = \mathcal{T}_k(x^{(p)} + \lambda d_{CN})$ or its OT counterpart. Establishes global convergence under RIP and provides computational complexity analyses; empirical experiments on random sparse problems demonstrate competitive performance in terms of success frequency and accuracy relative to state-of-the-art methods. These results suggest a scalable, Newton-inspired approach for sparse recovery with practical applicability in signal processing and related domains.

Abstract

Thresholding algorithms for sparse optimization problems involve two key components: search directions and thresholding strategies. In this paper, we use the compressed Newton direction as a search direction, derived by confining the classical Newton step to a low-dimensional subspace and embedding it back into the full space with diagonal regularization. This approach significantly reduces the computational cost for finding the search direction while maintaining the efficiency of Newton-like methods. Based on this new search direction, we propose two major classes of algorithms by adopting hard or optimal thresholding: the compressed Newton-direction-based thresholding pursuit (CNHTP) and compressed Newton-direction-based optimal thresholding pursuit (CNOTP). We establish the global convergence of the proposed algorithms under the restricted isometry property. Experimental results demonstrate that the proposed algorithms perform comparably to several state-of-the-art methods in terms of success frequency and solution accuracy for solving the sparse optimization problem.

Compressed Newton-direction-based Thresholding Methods for Sparse Optimization Problems

TL;DR

Addresses sparse optimization with by introducing a compressed Newton direction that operates in a low-dimensional subspace and is embedded into the full space via diagonal regularization. Builds two algorithm families, CNHT/CNHTP (hard/ pursuit) and CNOT/CNOTP (optimal thresholding), integrated into a unified iterative framework or its OT counterpart. Establishes global convergence under RIP and provides computational complexity analyses; empirical experiments on random sparse problems demonstrate competitive performance in terms of success frequency and accuracy relative to state-of-the-art methods. These results suggest a scalable, Newton-inspired approach for sparse recovery with practical applicability in signal processing and related domains.

Abstract

Thresholding algorithms for sparse optimization problems involve two key components: search directions and thresholding strategies. In this paper, we use the compressed Newton direction as a search direction, derived by confining the classical Newton step to a low-dimensional subspace and embedding it back into the full space with diagonal regularization. This approach significantly reduces the computational cost for finding the search direction while maintaining the efficiency of Newton-like methods. Based on this new search direction, we propose two major classes of algorithms by adopting hard or optimal thresholding: the compressed Newton-direction-based thresholding pursuit (CNHTP) and compressed Newton-direction-based optimal thresholding pursuit (CNOTP). We establish the global convergence of the proposed algorithms under the restricted isometry property. Experimental results demonstrate that the proposed algorithms perform comparably to several state-of-the-art methods in terms of success frequency and solution accuracy for solving the sparse optimization problem.

Paper Structure

This paper contains 12 sections, 9 theorems, 102 equations, 5 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Algorithms
  3. Compressed Newton search direction
  4. Compressed Newton-direction-based algorithms
  5. Theoretical analysis
  6. Theoretical results for CNHT and CNHTP
  7. Theoretical results for CNOT and CNOTP
  8. Computational complexity
  9. Numerical Investigations
  10. Sensitivity to parameter choice
  11. Phase transition curves
  12. Summary

Key Result

Lemma 3.1

foucart2013mathematical Let $A \in \mathbb{R}^{m \times n}$ satisfy the RIP of order $t$ with constant $\delta_t$. For any vector $v \in \mathbb{R}^n$ and any index set $\Lambda \subseteq [N]$, one has

Figures (5)

  • Figure 1: Performance comparison of CNHTP and CNOTP with $\lambda=1$, $q=k$, $\alpha=1$, and different values of $\gamma$ ($\gamma_1=0$, $\gamma_2=0.001$, $\gamma_3=0.01$, $\gamma_4=0.1$, $\gamma_5=0.3$, $\gamma_6=0.5$, $\gamma_7=0.7$, $\gamma_8=1$).
  • Figure 2: Performance comparison of CNHTP and CNOTP with $\lambda=1$, $\alpha=1$, $\gamma=0.01$ and different values of $q$ ($q_1=k, q_2=\lceil 1.25 k\rceil, q_3=\lceil 1.5 k\rceil$, $q_4= \lceil 1.75 k\rceil$, and $q_5 =2k$).
  • Figure 3: Performance comparison of CNHTP and CNOTP with parameters $q=k$, $\alpha=1$, $\gamma=0.01$ and different values of $\lambda$ ($\lambda=1,2,\dots, 7$).
  • Figure 4: Phase transition curves for sparse signal recovery with success frequency of 0.9.
  • Figure 5: Time consumption ratio NTROTP/CNOTP for successful recovery.

Theorems & Definitions (17)

  • Remark 1
  • Definition 3.1
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • proof
  • Theorem 3.4
  • proof
  • Lemma 3.5
  • Theorem 3.6
  • ...and 7 more