Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controlling Recurrent Neural Networks by Conceptors

Herbert Jaeger

TL;DR

A mechanism of neurodynamical organization, called conceptors, is proposed, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic, and helps explain how conceptual-level information processing emerges naturally and robustly in neural systems.

Abstract

The human brain is a dynamical system whose extremely complex sensor-driven neural processes give rise to conceptual, logical cognition. Understanding the interplay between nonlinear neural dynamics and concept-level cognition remains a major scientific challenge. Here I propose a mechanism of neurodynamical organization, called conceptors, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic. It becomes possible to learn, store, abstract, focus, morph, generalize, de-noise and recognize a large number of dynamical patterns within a single neural system; novel patterns can be added without interfering with previously acquired ones; neural noise is automatically filtered. Conceptors help explaining how conceptual-level information processing emerges naturally and robustly in neural systems, and remove a number of roadblocks in the theory and applications of recurrent neural networks.

Controlling Recurrent Neural Networks by Conceptors

TL;DR

A mechanism of neurodynamical organization, called conceptors, is proposed, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic, and helps explain how conceptual-level information processing emerges naturally and robustly in neural systems.

Abstract

The human brain is a dynamical system whose extremely complex sensor-driven neural processes give rise to conceptual, logical cognition. Understanding the interplay between nonlinear neural dynamics and concept-level cognition remains a major scientific challenge. Here I propose a mechanism of neurodynamical organization, called conceptors, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic. It becomes possible to learn, store, abstract, focus, morph, generalize, de-noise and recognize a large number of dynamical patterns within a single neural system; novel patterns can be added without interfering with previously acquired ones; neural noise is automatically filtered. Conceptors help explaining how conceptual-level information processing emerges naturally and robustly in neural systems, and remove a number of roadblocks in the theory and applications of recurrent neural networks.

Paper Structure

This paper contains 66 sections, 23 theorems, 211 equations, 49 figures, 1 table.

Table of Contents

  1. Overview
  2. Introduction
  3. Motivation
  4. Mathematical Preliminaries
  5. Theory and Demonstrations
  6. Networks and Signals
  7. Driving a Reservoir with Different Patterns
  8. Storing Patterns in a Reservoir, and Training the Readout
  9. Conceptors: Introduction and Basic Usage in Retrieval
  10. A Similarity Measure for Excited Network Dynamics
  11. Online Learning of Conceptor Matrices
  12. Morphing Patterns
  13. Understanding Aperture
  14. The Semantics of $\alpha$ as "Aperture"
  15. Aperture Adaptation and Final Definition of Conceptor Matrices
  16. ...and 51 more sections

Key Result

Proposition 1

Let $R = E[x\,x']$ be a correlation matrix and $\alpha \in (0,\infty)$. Then,

Figures (49)

  • Figure 1: Deriving conceptors from network dynamics. A. Network layout. Arrows indicate synaptic links. B. Driving the reservoir with four different input patterns. Left panels: 20 timesteps of input pattern $p(n)$ (black thin line) and conceptor-controlled output $y(n)$ (bold light gray). Second column: 20 timesteps of traces $x_i(n), x_j(n)$ of two randomly picked reservoir neurons. Third column: the singular values $\sigma_i$ of the reservoir state correlation matrix $R$ in logarithmic scale. Last column: the singular values $s_i$ of the conceptors $C$ in linear plotting scale. C. From pattern to conceptor. Left: plots of value pairs $x_i(n),x_j(n)$ (dots) of the two neurons shown in first row of B and the resulting ellipse with axis lengths $\sigma_1, \sigma_2$. Right: from $R$ (thin light gray) to conceptor $C$ (bold dark gray) by normalizing axis lengths $\sigma_1, \sigma_2$ to $s_1, s_2$.
  • Figure 2: Morphing between, and generalizing beyond, four loaded patterns. Each panel shows a 15-step autonomously generated pattern (plot range between $-1$ and $+1$). Panels with bold frames: the four loaded prototype patterns (same patterns as in Fig. \ref{['fig1Main']}B.)
  • Figure 3: Aperture adaptation for re-generating four chaotic attractors. A Lorenz attractor. Five versions re-generated with different apertures (values inserted in panels) and original attractor (green). B Best re-generations of the other three attractors (from left to right: Rössler, Mackey-Glass, and Hénon, originals in green). C Log10 of the attenuation criterion plotted against the log10 of aperture. Dots mark the apertures used for plots in A and B.
  • Figure 4: Boolean operations on conceptors. Red/blue (thin) ellipses represent source conceptors $C_1,C_2$. Magenta (thick) ellipses show $C_1 \vee C_2$, $C_1 \wedge C_2$, $\neg C_1$ (from left to right).
  • Figure 5: Incremental pattern storing in a neural memory. Each panel shows a 20-timestep sample of the correct training pattern $p^j$ (black line) overlaid on its reproduction (green line). The memory fraction used up until pattern $j$ is indicated by the panel fraction filled in red; the quota value is printed in the left bottom corner of each panel.
  • ...and 44 more figures

Theorems & Definitions (36)

  • Definition 1
  • Proposition 1
  • Proposition 2
  • Definition 2
  • Definition 3
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Definition 4
  • ...and 26 more