How to warm-start your unfolding network

TL;DR

The paper tackles improving deep unfolding networks for compressed sensing (CS), where the goal is to recover $x \in \mathbb{R}^n$ from $y = A x + e$ with $A \in \mathbb{R}^{m \times n}$ and $m < n$, assuming sparsity after transform $W$. It introduces C-DEC, a continuation-based warm-start of the overparameterized unfolding network $DECONET$, formed by integrating a continuation loop with the unfolded CS solver and training with the $\log\cosh$ loss. Empirical results on MNIST and CIFAR10 show that continuation and overparameterization yield smoother loss landscapes and lower test and generalization errors compared to DECONET, across various layer counts and sparsifying transforms. The work also visualizes loss landscapes and discusses potential theoretical analysis of continuation's impact and robustness, highlighting practical significance for robust CS reconstructions.

Abstract

We present a new ensemble framework for boosting the performance of overparameterized unfolding networks solving the compressed sensing problem. We combine a state-of-the-art overparameterized unfolding network with a continuation technique, to warm-start a crucial quantity of the said network's architecture; we coin the resulting continued network C-DEC. Moreover, for training and evaluating C-DEC, we incorporate the log-cosh loss function, which enjoys both linear and quadratic behavior. Finally, we numerically assess C-DEC's performance on real-world images. Results showcase that the combination of continuation with the overparameterized unfolded architecture, trained and evaluated with the chosen loss function, yields smoother loss landscapes and improved reconstruction and generalization performance of C-DEC, consistently for all datasets.

Figures (2)

• Figure 1: Performance plots of C-DEC, with $W\in\mathbb{R}^{10n\times n}$, on MNIST (left) and CIFAR10 (right) datasets.
• Figure 2: Loss landscapes of 5-layer C-DEC on (a) MNIST and (b) CIFAR10 datasets.