A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning

TL;DR

The paper investigates whether artificial systems spontaneously develop brain-like information integration by applying Integrated Information Decomposition ($\Phi$ID) to measure synergy and redundancy among information-processing units in large language models. It demonstrates a brain-like topology where middle layers form a synergistic core, while early and late layers are redundancy-dominated, and shows this structure emerges through learning rather than initialisation. Causally, ablating high-synergy components degrades performance more than random or redundant components, and reinforcement learning fine-tuning targeting the synergistic core yields larger gains than targeting redundant regions; supervised fine-tuning shows no such advantage. These findings suggest a convergent computational principle across biology and AI, with implications for principled design, generalisation, and testable predictions about brain organisation and plasticity.

Abstract

The independent evolution of intelligence in biological and artificial systems offers a unique opportunity to identify its fundamental computational principles. Here we show that large language models spontaneously develop synergistic cores -- components where information integration exceeds individual parts -- remarkably similar to those in the human brain. Using principles of information decomposition across multiple LLM model families and architectures, we find that areas in middle layers exhibit synergistic processing while early and late layers rely on redundancy, mirroring the informational organisation in biological brains. This organisation emerges through learning and is absent in randomly initialised networks. Crucially, ablating synergistic components causes disproportionate behavioural changes and performance loss, aligning with theoretical predictions about the fragility of synergy. Moreover, fine-tuning synergistic regions through reinforcement learning yields significantly greater performance gains than training redundant components, yet supervised fine-tuning shows no such advantage. This convergence suggests that synergistic information processing is a fundamental property of intelligence, providing targets for principled model design and testable predictions for biological intelligence.

Figures (5)

• Figure 1: Schematic diagram of the paper.(a) Methods: We measure the attention head activation of each attention heads in each transformer block, during the generation of a sequence of 100 tokens. Pairs of activation time series are considered as sources of information ($S_1$, $S_2$) and used to calculate the information-theoretic measures of synergy and redundancy. (b) Key result: LLMs contain a "synergistic core" in their middle layers, while their early and late layers (closer to input and output, respectively) are predominantly redundant. Through observational and interventional experiments, we demonstrate the relevance of the synergistic core for model behaviour and generalization.
• Figure 2: LLMs contain a synergistic core comprising their middle layers. (a) Heatmaps showing synergy and redundancy between pairs of attention heads in Gemma 3 4B. (b) Heatmap showing the synergy-redundancy ranks in Gemma 3 4B. High (red) values indicate that an attention head's interactions are predominantly synergistic, while low (blue) values indicate they are primarily redundant. (c) Synergistic cores across several LLMs, with the y-axis representing the synergy-redundancy rank. Layer numbers are normalised from $0$ to $1$, and the synergy-redundancy rank is normalised using min-max scaling to enable comparisons across models of different sizes. The middle layers of the models exhibit the most synergistic interactions. DeepSeek V2 Lite's synergistic core is computed at an expert level, whereas the rest of models are based on attention heads.
• Figure 3: LLM's synergistic core emerges through training and shares topological features with the human brain. (a) Normalised synergy–redundancy rank through training for Pythia-1B biderman2023pythia. (b) Synergistic and redundant cores for Gemma 3 1B Instruct gemma_2025 represented as undirected graphs. Only the top 10% strongest connections are shown for clarity. (c) Graph-theoretical properties of the synergy and redundancy matrices for different LLMs. For DeepSeek V2 Lite, the matrices are computed expert-wise, and for the remaining models they are computed attention-head-wise.
• Figure 4: Ablating the synergistic core causes greater changes in behaviour and drops in performance.(a) Behaviour divergence as a function of the fraction of nodes deactivated (experts for DeepSeek V2 Lite and attention heads for the remaining models), quantified as the KL divergence between the token strings of original and ablated models. Solid curves correspond to ablating nodes in synergistic order, while dashed curves correspond to randomly ordered ablating. The shaded regions around the dashed curves indicate the standard deviation across five random-order runs. (b) Comparison of accuracy on the MATH benchmark hendrycks2021measuringmathematicalproblemsolving when perturbing the synergistic core, the redundant core, or random subsets of each model. Perturbations were applied by injecting Gaussian noise into the query/output projections of selected attention heads, or into full expert parameters for MoE architectures.
• Figure 5: Fine-tuning the synergistic core yields better performance using reinforcement learning (but not supervised) fine-tuning. Comparison of MATH benchmark accuracy achieved by Qwen2.5-Math-1.5B after Supervised fine-tuning on OpenMathInstruct-2 (left) or Reinforcement Learning fine-tuning on the MATH training set (right), using three different training approaches: synergistic core, redundant core, and random subset training. Each training method is independently run five times, and the best-performing checkpoint from the last five evaluations (at steps 2500, 3000, 3500, 4000, 4500, and 5000) is selected for RL (for SFT, the final checkpoint coincided with the best). Under reinforcement learning (right), synergistic core training shows large improvements over random subset training (Hedges' $g\!\approx\!1.4$) and redundant core training (Hedges' $g\!\approx\!5.0$), while no significant differences are observed under supervised fine-tuning (left). Asterisks indicate statistical significance: *$p < 0.05$, **$p < 0.01$, and ***$p < 0.001$ (n.s. represents "not significant", $p > 0.05$).