High-dimensional dynamics in low-dimensional networks

TL;DR

It is shown that several low-rank network structures arising in nature satisfy the conditions for generating high-dimensional dynamics and low-rank suppression and the results clarify important, but counterintuitive relationships between a recurrent network's connectivity structure and the structure of the dynamics it generates.

Abstract

Many networks in nature and applications have an approximate low-rank structure in the sense that their connectivity structure is dominated by a few dimensions. It is natural to expect that dynamics on such networks would also be low-dimensional. Indeed, theoretical results show that low-rank networks produce low-dimensional dynamics whenever the network is isolated from external perturbations or input. However, networks in nature are rarely isolated. Here, we study the dimensionality of dynamics in recurrent networks with low-dimensional structure driven by high-dimensional inputs or perturbations. We find that dynamics in such networks can be high- or low-dimensional and we derive novel, precise conditions on the network structure under which dynamics are high-dimensional. In many low-rank networks, dynamics are suppressed in directions aligned with the network's low-rank structure, a phenomenon we term ``low-rank suppression.'' We show that several low-rank network structures arising in nature satisfy the conditions for generating high-dimensional dynamics and low-rank suppression. Our results clarify important, but counterintuitive relationships between a recurrent network's connectivity structure and the structure of the dynamics it generates.

Figures (19)

• Figure 1: Response properties of a recurrent network with rank-one structure.a) Schematic of model: The connectivity matrix, $W$, quantifies connections between nodes, ${\bf{z}}$, which receive external perturbations or input, ${\bf{x}}$. b) The singular values of $W$ have one dominant term, indicating approximate rank-one structure. c) The eigenvalues of $W$ have a dominant, negative term. d) The distribution of variance across principal components of a Gaussian stochastic input (${\bf{x}}(t)$; green) and the response (${\bf{z}}(t)$; blue). e,f) The network response (e) and input (f) projected onto the plane determined by ${\bf{u}}$ and a random vector, ${\bf{u}}_\textrm{rand}$, demonstrates low-rank suppression along ${\bf{u}}$. g) The network response (${\bf{z}}(t)$, top) and its norm ($\|{\bf{z}}(t)\|$, bottom) given an input aligned to the low-rank structure of the network (${\bf{x}}_\textrm{aligned}$) and a random input (${\bf{x}}_\textrm{rand}$). h) Local network input (purple) cancels with external input (green) to produce suppressed network responses (blue) in the direction of ${\bf{u}}$.
• Figure 2: Response properties of a non-normal recurrent network with rank-one structure. Same as Figure \ref{['F1']} except the low-rank part of $W$ is non-normal. The network still exhibits low-rank suppression and cancellation (f, g, h) even in the absence of strong negative eigenvalues (c). The network exhibits low-dimensional dynamics (d), in contrast to Figure \ref{['F1']}.
• Figure 3: Conditions for high-dimensional dynamics in a network with rank-two structure.a) Singular values of $W$ demonstrate an effective low-rank structure. b) The alignment matrix, $P$, when $W_0$ is normal and symmetric. c) The variance explained by each principal component of the network dynamics, ${\bf{z}}(t)$. d) The eigenvalues of $W$. e--g) Same as b--d except $W_0$ is EP, but non-normal. h--j) Same as b--d except $W_0$ is not EP. k--m) Same as h--j but using a different non-EP structure. n--p) Same as b--d except $W_0$ is non-EP with a non-vanishing EP component. See the text for the precise definition of $W_0$ in each case.
• Figure 4: Low-rank suppression and excitatory-inhibitory balance in a modular network.a,b) A modular network with biased blocks modeling excitatory and inhibitory neurons has low-rank structure. c) The alignment matrix shows that the network is EP, but not normal. d) An input that is constant within each block (red) is aligned to the low-rank part, but a random input (gray) is not. e) Response magnitude is suppressed for the aligned input relative to the random input. f) The excitatory (positive; red) component of an input balances with the inhibitory (negative; blue) component to produce a much smaller total (gray) component, a widely observed phenomenon in neural circuits.
• Figure 5: Low-rank suppression in a network with spatial structure.a) Connectivity matrix, $W$. Connection strength is a Gaussian function of distance. b) Singular values of $W$ demonstrate effective low-rank structure. c) A spatially smooth perturbation (red) is aligned to the low-rank part of $W$ while a spatially disordered perturbation (gray) is not. d,e) Therefore, the network response to a smooth perturbation is much weaker than the response to a disordered perturbation (compare vertical tick marks).

Theorems & Definitions (5)

• Claim 1
• Claim 2
• Claim 2
• Claim 2
• proof