A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions

TL;DR

This work introduces infomorphic neurons, a two-input-class neural unit whose local learning objective is derived from Partial Information Decomposition (PID). By decomposing a neuron's output information into unique, redundant, and synergistic components, the authors define a parametric local goal function that can be optimized with analytic gradients. They demonstrate the framework across three learning paradigms—supervised, unsupervised, and online associative memory—showing that PID-based local goals enable interpretable and flexible learning dynamics and, in some cases, competitive performance relative to traditional methods. The results suggest that information-theoretic local goals can bridge biological and artificial learning, offering a principled, interpretable foundation for self-organizing networks with potential for scalable, task-agnostic learning rules. The work also outlines avenues for improving biological plausibility and extending the approach to deeper architectures and more complex input interactions, leveraging the PID framework to study and design local learning dynamics.

Abstract

Despite the impressive performance of biological and artificial networks, an intuitive understanding of how their local learning dynamics contribute to network-level task solutions remains a challenge to this date. Efforts to bring learning to a more local scale indeed lead to valuable insights, however, a general constructive approach to describe local learning goals that is both interpretable and adaptable across diverse tasks is still missing. We have previously formulated a local information processing goal that is highly adaptable and interpretable for a model neuron with compartmental structure. Building on recent advances in Partial Information Decomposition (PID), we here derive a corresponding parametric local learning rule, which allows us to introduce 'infomorphic' neural networks. We demonstrate the versatility of these networks to perform tasks from supervised, unsupervised and memory learning. By leveraging the interpretable nature of the PID framework, infomorphic networks represent a valuable tool to advance our understanding of the intricate structure of local learning.

Figures (11)

• Figure 1: The infomorphic neuron model, analogous to cortical pyramidal neurons, separately integrates two distinct classes of inputs. The neuron adjusts its synaptic weights to maximize the local goal function $G$, based on an information-theoretic decomposition of its own output information. (A) Cortical pyramidal neurons with separate synaptic integration sites for basal and apical dendrites, the former driving output and the latter providing contextual modulation. (B) The infomorphic neuron, modeled after these neurons, is characterized by two functionally distinct sets of inputs that are scaled by synaptic weights and added to obtain the integrated input signals $R$ (receptive) and $C$ (contextual). R and C contribute individually to the probabilities of the neuron's binary output, which are computed using an activation function $A(R,C)$ and a sigmoid transformation function. (C) The total Shannon output information $H(Y)$ of the neuron consists of the mutual information with the joint inputs $I(Y : R, C)$ and additional residual information $H_{\mathop{\mathrm{res}}\nolimits} = H(Y \mid R, C)$ that originates directly from the stochasticity of the neuron. Using Partial Information Decomposition (PID), the joint mutual information $I(Y : R, C)$ can be further subdivided into four information contributions: (i) $I_{\mathop{\mathrm{red}}\nolimits}$, the redundant information that is provided by either $R$ or $C$ individually, (ii) $I_{\mathop{\mathrm{unq}}\nolimits, R}$, the unique information of $R$ that is only provided by $R$ but not by $C$, (iii) $I_{\mathop{\mathrm{unq}}\nolimits, C}$, the unique information of $C$ that is only provided by $C$ but not by $R$, and (iv) $I_{\mathop{\mathrm{syn}}\nolimits}$, the synergistic information that is provided by $R$ and $C$ only when taken jointly but neither by $R$ nor $C$ taken individually. (D) Any classical mutual-information-based decomposition can only provide a linear combination of the underlying PID quantities. (D Upper) Classical decomposition into four information contributions that formed the basis for prior work kay1994informationkay1997activationkay1999neuralkay2011coherent: the co-information $I(Y: R : C)$, the two conditional mutual information values $I(Y: R\mid C)$ and $I(Y: C\mid R)$, and the stochasticity-caused residual entropy $H_{\mathop{\mathrm{res}}\nolimits}$. (D Lower) The five contributions that are quantified using PID. (E) The neuron's synaptic weights $\mathbf{w}$ are optimized to maximize a goal function $G$ that is based on the Partial Information Decomposition of the neuron's overall output information $H(Y)$ and parameterised by $\mathbf{\Gamma} = (\Gamma_{\mathop{\mathrm{unq}}\nolimits, R}, \Gamma_{\mathop{\mathrm{unq}}\nolimits, C}, \Gamma_{\mathop{\mathrm{red}}\nolimits}, \Gamma_{\mathop{\mathrm{syn}}\nolimits}, \Gamma_{\mathop{\mathrm{res}}\nolimits})$. Panel (A) is adapted from Fabian Mikulasch's original depiction of a pyramidal neuron mikulasch2023error (CC0).
• Figure 2: In supervised learning, high redundant information $I_{\mathop{\mathrm{red}}\nolimits}$ and low unique receptive information $I_{\mathop{\mathrm{unq}}\nolimits,R}$ are associated with high classification accuracy.(Left) Time evolution of information quantities for each neuron in a single, randomly chosen network performing supervised learning. Neuron $i$ corresponds to label $i$. The dashed line shows the empirical entropy of the binary one-vs-all distribution of each label in the test data set. (Top right) The confusion matrix for this network, after applying a winner-take-all readout. (Bottom right) The firing probability of each neuron for each digit, averaged over test images.
• Figure 3: The receptive fields of supervised infomorphic learning are similar to those of one-vs-all logistic regression. First and third column: Receptive fields of all neurons after training in a randomly chosen supervised infomorphic network. Second and fourth column: Corresponding receptive fields obtained from one-vs-all logistic regression with vanilla gradient descent. The depicted receptive fields are centered at a weight of $w_R = 0$ and re-scaled to the interval $[-1,1]$. $R$-values indicate cosine similarity between corresponding receptive fields. The colored vectors bordering the receptive fields of logistic regression indicate row-by-row and column-by-column cosine similarity between corresponding receptive fields. Note that all values are on the same scale, indicated by the color bar at the bottom.
• Figure 4: The test accuracy of supervised infomorphic learning is similar using $P(Y\mid r,0)$ or $P(Y\mid r)$. The average winner-take-all test accuracy using $P(Y\mid r,0)$ and $P(Y\mid r)$ across 100 training runs, with both accuracies approaching that of logistic regression (reaching on average $89.7\%$ vs. $91.9\%$ for log. regr.). Note that the 95-percentile is being displayed.
• Figure 5: Infomorphic neurons learn to encode distinct input features by unsupervised maximization of $I_{\mathop{\mathrm{unq}}\nolimits,R}$ . The receptive fields $\mathbf{w}_R$ and contextual fields $\mathbf{w}_C$ (first and third columns), and the evolution of all information contributions over learning (second and fourth columns) are shown for each neuron of a randomly chosen network successfully performing unsupervised learning. The receptive input consists of 8 horizontal bars in an 8-by-8 grid, each bar appearing with probability $p = 0.5$. The contextual input is a vector of length 7, transmitting the output from all other neurons, without self-connections. Note that the values of all receptive and contextual weights are on the same scale indicated at the bottom of the figure.