Quantifying information stored in synaptic connections rather than in firing patterns of neural networks

TL;DR

A theoretical framework using continuous Hopfield networks as an exemplar for associative neural networks, and data that follow mixtures of broadly applicable multivariate log-normal distributions is developed, revealing synergistic interactions among synapses.

Abstract

A cornerstone of our understanding of both biological and artificial neural networks is that they store information in the strengths of connections among the constituent neurons. However, in contrast to the well-established theory for quantifying information encoded by the firing patterns of neural networks, little is known about quantifying information encoded by its synaptic connections. Here, we develop a theoretical framework using continuous Hopfield networks as an exemplar for associative neural networks, and data that follow mixtures of broadly applicable multivariate log-normal distributions. Specifically, we analytically derive the Shannon mutual information between the data and singletons, pairs, triplets, quadruplets, and arbitrary n-tuples of synaptic connections within the network. Our framework corroborates well-established insights about storage capacity of, and distributed coding by, neural firing patterns. Strikingly, it discovers synergistic interactions among synapses, revealing that the information encoded jointly by all the synapses exceeds the 'sum of its parts'. Taken together, this study introduces an interpretable framework for quantitatively understanding information storage in neural networks, one that illustrates the duality of synaptic connectivity and neural population activity in learning and memory.

Figures (4)

• Figure 1: Model setup. (A) Real-world distribution modelled as several independent patterns following multivariate log-normal distribution. (B) Continuous Hopfield network with an example synaptic ensemble $\mathbf{w}$ (red) composed of four connections.
• Figure 2: Information and the number of data patterns. For each $N$, the curves show how the information encoded in a single example weight varies with (A) $\tilde{\sigma}$ and (B) $\tilde{\rho}$. As $N$ increases, the overall curves shift downward, indicating a decrease in the total information encoded. This decreasing trend is significant for randomly sampled distribution configurations with (C) the same covariance matrix and (D) different covariance matrices.
• Figure 3: Information and the number of synaptic connections in an ensemble. For each $n$, the curves show how the information encoded in an example ensemble varies with (A) $\tilde{\sigma}$ and (B) $\tilde{\rho}$. As $n$ increases, the overall curves shift upward, indicating an increase in the total information encoded. This increasing trend also holds significant for randomly sampled distribution configurations with (C) the same covariance matrix and (D) different covariance matrices.
• Figure 4: Information contributed per synaptic connection as a function of ensemble size. (A) The relationship between $n$ and the mean (green, left axis) as well as standard deviation (blue, right axis) of $\ln(MI/n)$. (B) Histograms showing the distribution of $\ln(MI/n)$ for $n = 1, 4, 9, 16$.