Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discovering Sparse Recovery Algorithms Using Neural Architecture Search

Patrick Yubeaton, Sarthak Gupta, M. Salman Asif, Chinmay Hegde

TL;DR

This work addresses the challenge of designing sparse-recovery algorithms by automating algorithm discovery with Neural Architecture Search (NAS). By treating iterative solvers as unrolled networks, the authors attempt to rediscover ISTA and its accelerated variant FISTA within a large NAS search space and demonstrate that NAS can recover proximal operators like soft-thresholding. They further show methods to reduce NAS training time and explore learning data-distribution–specific proximal projections, indicating NAS can adapt proximal mappings to different data regimes. The findings suggest a data-driven, automated pathway to synthesize efficient sparse solvers beyond traditional ISTA/FISTA, with potential applicability to broader inverse-problem settings. The work lays groundwork for automated algorithm design in signal processing and meta-learning, proposing practical avenues for faster NAS and data-tailored proximal operators.

Abstract

The design of novel algorithms for solving inverse problems in signal processing is an incredibly difficult, heuristic-driven, and time-consuming task. In this short paper, we the idea of automated algorithm discovery in the signal processing context through meta-learning tools such as Neural Architecture Search (NAS). Specifically, we examine the Iterative Shrinkage Thresholding Algorithm (ISTA) and its accelerated Fast ISTA (FISTA) variant as candidates for algorithm rediscovery. We develop a meta-learning framework which is capable of rediscovering (several key elements of) the two aforementioned algorithms when given a search space of over 50,000 variables. We then show how our framework can apply to various data distributions and algorithms besides ISTA/FISTA.

Discovering Sparse Recovery Algorithms Using Neural Architecture Search

TL;DR

This work addresses the challenge of designing sparse-recovery algorithms by automating algorithm discovery with Neural Architecture Search (NAS). By treating iterative solvers as unrolled networks, the authors attempt to rediscover ISTA and its accelerated variant FISTA within a large NAS search space and demonstrate that NAS can recover proximal operators like soft-thresholding. They further show methods to reduce NAS training time and explore learning data-distribution–specific proximal projections, indicating NAS can adapt proximal mappings to different data regimes. The findings suggest a data-driven, automated pathway to synthesize efficient sparse solvers beyond traditional ISTA/FISTA, with potential applicability to broader inverse-problem settings. The work lays groundwork for automated algorithm design in signal processing and meta-learning, proposing practical avenues for faster NAS and data-tailored proximal operators.

Abstract

The design of novel algorithms for solving inverse problems in signal processing is an incredibly difficult, heuristic-driven, and time-consuming task. In this short paper, we the idea of automated algorithm discovery in the signal processing context through meta-learning tools such as Neural Architecture Search (NAS). Specifically, we examine the Iterative Shrinkage Thresholding Algorithm (ISTA) and its accelerated Fast ISTA (FISTA) variant as candidates for algorithm rediscovery. We develop a meta-learning framework which is capable of rediscovering (several key elements of) the two aforementioned algorithms when given a search space of over 50,000 variables. We then show how our framework can apply to various data distributions and algorithms besides ISTA/FISTA.
Paper Structure (14 sections, 2 equations, 4 figures)

This paper contains 14 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background
  3. ISTA and FISTA
  4. Learning to Optimize (L2O).
  5. Neural Architecture Search
  6. Our Results
  7. Recovering the soft thresholding operator
  8. Dataset
  9. Model Architecture
  10. Training Hyperparameters
  11. Decreasing the Training Time of NAS
  12. Can NAS learn proximal projections tailored to specific data distributions?
  13. Dataset
  14. Discussion

Figures (4)

  • Figure 1: (left) The ISTA algorithm represented via a nonlinear dynamical system, or an RNN; here, $W = I - \eta A^T A$. (right) Unfolding the RNN to $K$ stages. Our DISCO approach proposes to automatically learn the edges and weights of this recurrent neural network using NAS techniques.
  • Figure 2: We show the NAS weights for each operation during the model training. The NAS weights are obtained by taking the softmax of the NAS $\alpha$ values for each layer and adding up the values for each operation. We then normalize the values to be between 0 and 1.
  • Figure 3: We show the NAS weights for each operation during model training. The NAS weights are obtained by taking the softmax of the NAS $\alpha$ values for the one layer model and normalizing the values to be between 0 and 1. Figure best viewed in color.
  • Figure 4: We show the NAS weights for each operation during model training. The NAS weights are obtained by taking the softmax of the NAS $\alpha$ values for the one layer model and normalizing the values to be between 0 and 1.