Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discriminative reconstruction via simultaneous dense and sparse coding

Abiy Tasissa, Emmanouil Theodosis, Bahareh Tolooshams, Demba Ba

Abstract

Discriminative features extracted from the sparse coding model have been shown to perform well for classification. Recent deep learning architectures have further improved reconstruction in inverse problems by considering new dense priors learned from data. We propose a novel dense and sparse coding model that integrates both representation capability and discriminative features. The model studies the problem of recovering a dense vector $\mathbf{x}$ and a sparse vector $\mathbf{u}$ given measurements of the form $\mathbf{y} = \mathbf{A}\mathbf{x}+\mathbf{B}\mathbf{u}$. Our first analysis relies on a geometric condition, specifically the minimal angle between the spanning subspaces of matrices $\mathbf{A}$ and $\mathbf{B}$, which ensures a unique solution to the model. The second analysis shows that, under some conditions on $\mathbf{A}$ and $\mathbf{B}$, a convex program recovers the dense and sparse components. We validate the effectiveness of the model on simulated data and propose a dense and sparse autoencoder (DenSaE) tailored to learning the dictionaries from the dense and sparse model. We demonstrate that (i) DenSaE denoises natural images better than architectures derived from the sparse coding model ($\mathbf{B}\mathbf{u}$), (ii) in the presence of noise, training the biases in the latter amounts to implicitly learning the $\mathbf{A}\mathbf{x} + \mathbf{B}\mathbf{u}$ model, (iii) $\mathbf{A}$ and $\mathbf{B}$ capture low- and high-frequency contents, respectively, and (iv) compared to the sparse coding model, DenSaE offers a balance between discriminative power and representation.

Discriminative reconstruction via simultaneous dense and sparse coding

Abstract

Discriminative features extracted from the sparse coding model have been shown to perform well for classification. Recent deep learning architectures have further improved reconstruction in inverse problems by considering new dense priors learned from data. We propose a novel dense and sparse coding model that integrates both representation capability and discriminative features. The model studies the problem of recovering a dense vector and a sparse vector given measurements of the form . Our first analysis relies on a geometric condition, specifically the minimal angle between the spanning subspaces of matrices and , which ensures a unique solution to the model. The second analysis shows that, under some conditions on and , a convex program recovers the dense and sparse components. We validate the effectiveness of the model on simulated data and propose a dense and sparse autoencoder (DenSaE) tailored to learning the dictionaries from the dense and sparse model. We demonstrate that (i) DenSaE denoises natural images better than architectures derived from the sparse coding model (), (ii) in the presence of noise, training the biases in the latter amounts to implicitly learning the model, (iii) and capture low- and high-frequency contents, respectively, and (iv) compared to the sparse coding model, DenSaE offers a balance between discriminative power and representation.

Paper Structure

This paper contains 28 sections, 9 theorems, 30 equations, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Organization of paper
  3. Notation
  4. Related work
  5. Compressive sensing from union of dictionaries
  6. Error correction
  7. Weighted Lasso
  8. Morphological component analysis (MCA)
  9. Dictionary learning and unrolling
  10. Contribution
  11. Technical background
  12. Compressive sensing with exact measurements
  13. Compressive sensing with structured random matrices
  14. Theoretical Analysis
  15. Feasibility problem
  16. ...and 13 more sections

Key Result

Theorem 1

Let $\mathbf{B}=\frac{1}{\sqrt{m}}\sum_{i=1}^{m} \mathbf{e}_i\mathbf{b}_i^T$ be a measurement matrix and let $\mathbf{u}$ be a fixed but otherwise arbitrary $s$-sparse vector in $\mathbb{R}^{n}$. Then with probability at least $1 - \frac{5}{n} -e^{-\beta}$ for some positive constant $\beta$, $\mathb provided that $m\ge C_\beta \mu(F)s\log n$. The constant $C_{\beta}=C_0(1+\beta)$ where $C_0$ is so

Figures (13)

  • Figure 1: A road-map of the main theoretical results in the manuscript.
  • Figure 2: Flow chart that shows the construction of the measurement matrices.
  • Figure 3: Phase transition curves for $p = 0.1m$ (left) and $p = 0.5m$ (right). Colors represent the probability of successful recovery, ranging from black (vector recovery failed in all trials) to yellow (recovery was always successful).
  • Figure 4: Normalized recovery error of $\bm{u}$ as the SNR varies (lower is better).
  • Figure 5: Decomposition of an MNIST image to its sparse and dense components. We visualize the minimization of the terms in \ref{['eq:all_min']}: the input $\mathbf{y}$ is adequately reconstructed (left), the component $\mathbf{A}\mathbf{x}$ is smooth relative to $\mathbf{B}\mathbf{u}$ (middle), and $\mathbf{u}$ is sparse.
  • ...and 8 more figures

Theorems & Definitions (17)

  • Theorem 1: candes2011probabilistic
  • Definition 1: kueng2014ripless
  • Theorem 2: kueng2014ripless
  • Lemma 1: kueng2014ripless,candes2011probabilistic
  • Theorem 3
  • proof
  • Definition 2
  • Theorem 4
  • proof
  • Corollary 5
  • ...and 7 more