Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Positive Competitive Networks for Sparse Reconstruction

Veronica Centorrino, Anand Gokhale, Alexander Davydov, Giovanni Russo, Francesco Bullo

TL;DR

This paper presents two continuous-time firing-rate networks, FCN and PFCN, to solve sparse reconstruction with and without nonnegativity constraints. By leveraging proximal operators, equilibria of the networks are shown to coincide with optimal solutions of the corresponding SR problems, and contraction theory yields rigorous convergence guarantees. The PFCN, in particular, is proved to be a positive system and, under RIP on the dictionary, to exhibit linear-exponential convergence to the SR optimum, with explicit rates and crossing times. Numerical experiments validate nonnegativity-preserving dynamics and competitive recovery performance against traditional LCA-based approaches. The work also provides a broad contractivity framework, explicit log-norm calculations, and proximal-operator tools to interpret these networks as biologically plausible solvers for structured optimization problems.

Abstract

We propose and analyze a continuous-time firing-rate neural network, the positive firing-rate competitive network (\pfcn), to tackle sparse reconstruction problems with non-negativity constraints. These problems, which involve approximating a given input stimulus from a dictionary using a set of sparse (active) neurons, play a key role in a wide range of domains, including for example neuroscience, signal processing, and machine learning. First, by leveraging the theory of proximal operators, we relate the equilibria of a family of continuous-time firing-rate neural networks to the optimal solutions of sparse reconstruction problems. Then, we prove that the \pfcn is a positive system and give rigorous conditions for the convergence to the equilibrium. Specifically, we show that the convergence: (i) only depends on a property of the dictionary; (ii) is linear-exponential, in the sense that initially the convergence rate is at worst linear and then, after a transient, it becomes exponential. We also prove a number of technical results to assess the contractivity properties of the neural dynamics of interest. Our analysis leverages contraction theory to characterize the behavior of a family of firing-rate competitive networks for sparse reconstruction with and without non-negativity constraints. Finally, we validate the effectiveness of our approach via a numerical example.

Positive Competitive Networks for Sparse Reconstruction

TL;DR

This paper presents two continuous-time firing-rate networks, FCN and PFCN, to solve sparse reconstruction with and without nonnegativity constraints. By leveraging proximal operators, equilibria of the networks are shown to coincide with optimal solutions of the corresponding SR problems, and contraction theory yields rigorous convergence guarantees. The PFCN, in particular, is proved to be a positive system and, under RIP on the dictionary, to exhibit linear-exponential convergence to the SR optimum, with explicit rates and crossing times. Numerical experiments validate nonnegativity-preserving dynamics and competitive recovery performance against traditional LCA-based approaches. The work also provides a broad contractivity framework, explicit log-norm calculations, and proximal-operator tools to interpret these networks as biologically plausible solvers for structured optimization problems.

Abstract

We propose and analyze a continuous-time firing-rate neural network, the positive firing-rate competitive network (\pfcn), to tackle sparse reconstruction problems with non-negativity constraints. These problems, which involve approximating a given input stimulus from a dictionary using a set of sparse (active) neurons, play a key role in a wide range of domains, including for example neuroscience, signal processing, and machine learning. First, by leveraging the theory of proximal operators, we relate the equilibria of a family of continuous-time firing-rate neural networks to the optimal solutions of sparse reconstruction problems. Then, we prove that the \pfcn is a positive system and give rigorous conditions for the convergence to the equilibrium. Specifically, we show that the convergence: (i) only depends on a property of the dictionary; (ii) is linear-exponential, in the sense that initially the convergence rate is at worst linear and then, after a transient, it becomes exponential. We also prove a number of technical results to assess the contractivity properties of the neural dynamics of interest. Our analysis leverages contraction theory to characterize the behavior of a family of firing-rate competitive networks for sparse reconstruction with and without non-negativity constraints. Finally, we validate the effectiveness of our approach via a numerical example.
Paper Structure (22 sections, 16 theorems, 39 equations, 5 figures, 1 table)

This paper contains 22 sections, 16 theorems, 39 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Mathematical Preliminaries
  3. Notation
  4. The Sparse Reconstruction Problems
  5. Norms and Logarithmic Norms
  6. Contraction Theory for Dynamical Systems
  7. Main Results
  8. Firing-rate Neural Networks For Solving Sparse Reconstruction Problems
  9. Analysis of the Proposed Networks
  10. Relating the FCN and PFCN with SR Problems
  11. Convergence Analysis
  12. Simulations
  13. Methods
  14. Proof of Theorem \ref{['thm:GLin-ExpS_of_pfrlcn']}
  15. Conclusions
  16. ...and 7 more sections

Key Result

Lemma II.1

Consider the FNN eq:firing_rate_nn with symmetric weight matrix $W$, and with activation function $\Psi$ being Lipschitz and slope restricted in $[0,1]$. If $\alpha(W) = 1$, then the FNN is weakly infinitesimally contracting with respect to some weighted Euclidean norm, say $\|\cdot\|_{2,D}$.The exp

Figures (5)

  • Figure 1: Strongly infinitesimally contracting systems: a) the distance between any two trajectories converges exponentially to the unique equilibrium point $x^{\star}$. Illustration of the dichotomy property of weakly contracting systems: b) the system has no equilibrium and every trajectory is unbounded or c) there exists at least one equilibrium and every trajectory is bounded. Images reused with permission from FB:23-CTDS.
  • Figure 2: The visual sensory input data $u \in \mathbb{R}^m$ is encoded by the receptive fields of simple cells in the mammalian visual cortex (V1) using only a small fraction of active (sparse) neurons. Formally, b) the input $u$ is reconstructed by a linear combination of an overcomplete ($n \gg m$) set of features $\Phi_i \in \mathbb{R}^n$ and sparse neurons $y \in \mathbb{R}^n$. c) Block scheme of the proposed (positive) firing-rate competitive network. The hidden node $x_i$ receives as stimulus the similarity score between the input signal $u\in \mathbb{R}^m$ and the dictionary element $\Phi_i\in \mathbb{R}^n$ and collectively all hidden neurons give as output a sparse (non-negative) vector $y = x \in \mathbb{R}^n$. Images reused with permission from FB:23-CTDS.
  • Figure 3: Schematic diagram summarizing the main results and their assumptions. With Theorem \ref{['thm:GLin-ExpS_of_pfrlcn']} we show that the PFCN \ref{['eq:pfr-lca_x_dot']} exhibits linear-exponential convergence towards the optimal solution of the positive SR problem \ref{['eq:positive_E_lasso_unconstrained']}. The result follows from: (i) establishing a link between the optimal solution of \ref{['eq:positive_E_lasso_unconstrained']} and the equilibria of \ref{['eq:pfr-lca_x_dot']}; (ii) characterizing contractivity of \ref{['eq:pfr-lca_x_dot']}.
  • Figure 4: Time evolution of the state/neuron variables of the proposed PFCN \ref{['eq:pfr-lca_x_dot']} (leftward panel) and of the LCA \ref{['eq:lca-soft-xi']} (rightward panel) networks. The cross symbols are the non-zero elements of the sparse vector $y_0$. Both the PFCN and the LCA converge to an equilibrium that is close to $y_0$. Note that, in accordance with Lemma \ref{['lem:positive_frcn']}, the state variables of the PFCN never become negative.
  • Figure 5: Trajectories of two randomly chosen nodes of the PFCN \ref{['eq:pfr-lca_x_dot']} from the active (leftward panel) and inactive (rightward panel) set in the planes defined by these two nodes, respectively. In the panels, the evolution is shown from $20$ randomly chosen initial conditions. In accordance with our results, the trajectories of the active neurons converge to positive values, while the trajectories of the inactive nodes converge to the origin.

Theorems & Definitions (28)

  • Definition 1: $k$-sparse vector and RIP EJC-TT:07
  • Definition 2: Equivalence ratio between two norms
  • Definition 3: Contracting systems FB:23-CTDS
  • Lemma II.1: Weakly contractivity of the FNN
  • Lemma III.1: Linking the optimal solutions of the SR problem and the equilibria of the FCN
  • Corollary III.2: Linking the optimal solutions of the positive SR problem and the equilibria of the PFCN
  • Definition 4: Active and inactive neuron
  • Remark 1
  • Remark 2
  • Theorem III.3: Global weak contractivity of the PFCN
  • ...and 18 more