Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Connectomes: A Generational Approach to Data-Efficient Language Models

Klemen Kotar, Greta Tuckute

TL;DR

This work introduces a generational learning paradigm that combines an outer evolutionary loop with an inner developmental loop, using a sparse connectome as a transmitted initialization to transfer learned priors across generations. The connectome is produced by iterative pruning of a large-dataset model until only $25\%$ of weights remain, then used to train a language model on a developmental dataset, with two control baselines for comparison. Across NLP benchmarks and brain/behavior alignment tasks, the Connectome model outperforms controls and approaches the performance of a dense model trained on a large corpus, demonstrating data efficiency and biological plausibility. The findings suggest that sparse, binary initializations distilled from large-scale data can substantially narrow the gap between single-generation models and biologically evolved networks, with potential for scalable, data-efficient language learning.

Abstract

Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks undergo a single, large training procedure without inherited constraints. In this preliminary work, we propose a framework that incorporates this crucial generational dimension - an "outer loop" of evolution that shapes the "inner loop" of learning - so that artificial networks better mirror the effects of evolution and individual learning in biological organisms. Focusing on language, we train a model that inherits a "model connectome" from the outer evolution loop before exposing it to a developmental-scale corpus of 100M tokens. Compared with two closely matched control models, we show that the connectome model performs better or on par on natural language processing tasks as well as alignment to human behavior and brain data. These findings suggest that a model connectome serves as an efficient prior for learning in low-data regimes - narrowing the gap between single-generation artificial models and biologically evolved neural networks.

Model Connectomes: A Generational Approach to Data-Efficient Language Models

TL;DR

This work introduces a generational learning paradigm that combines an outer evolutionary loop with an inner developmental loop, using a sparse connectome as a transmitted initialization to transfer learned priors across generations. The connectome is produced by iterative pruning of a large-dataset model until only of weights remain, then used to train a language model on a developmental dataset, with two control baselines for comparison. Across NLP benchmarks and brain/behavior alignment tasks, the Connectome model outperforms controls and approaches the performance of a dense model trained on a large corpus, demonstrating data efficiency and biological plausibility. The findings suggest that sparse, binary initializations distilled from large-scale data can substantially narrow the gap between single-generation models and biologically evolved networks, with potential for scalable, data-efficient language learning.

Abstract

Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks undergo a single, large training procedure without inherited constraints. In this preliminary work, we propose a framework that incorporates this crucial generational dimension - an "outer loop" of evolution that shapes the "inner loop" of learning - so that artificial networks better mirror the effects of evolution and individual learning in biological organisms. Focusing on language, we train a model that inherits a "model connectome" from the outer evolution loop before exposing it to a developmental-scale corpus of 100M tokens. Compared with two closely matched control models, we show that the connectome model performs better or on par on natural language processing tasks as well as alignment to human behavior and brain data. These findings suggest that a model connectome serves as an efficient prior for learning in low-data regimes - narrowing the gap between single-generation artificial models and biologically evolved neural networks.
Paper Structure (22 sections, 1 equation, 2 figures)

This paper contains 22 sections, 1 equation, 2 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Model development
  4. Model architecture
  5. Training datasets
  6. Generational outer loop: Wiring up the Connectome
  7. Developmental inner loop: Learning language representations
  8. Model evaluation
  9. Results
  10. The connectome model outperforms control models on NLP benchmarks
  11. Scaling up: Exploratory analysis
  12. The Connectome model aligns better or on par with human reading times and brain responses
  13. Limitations & Discussion
  14. Appendix
  15. Model architecture
  16. ...and 7 more sections

Figures (2)

  • Figure 1: A. Conceptual overview, see description in Sections \ref{['section:intro']} and \ref{['methods']}. B. Performance evaluation on standard NLP benchmarks: FineWeb validation loss (panel i), HellaSwag and MMLU (panel ii). C. Alignment with human reading times on naturalistic stories. D. Model-brain alignment, flexibly mapping all units within each model layer to brain responses (panel i) or through a more stringent procedure which localizes language-selective model units (panel ii). To ensure robustness of our results, all analyses are conducted with four seeds per model instantiation, and plots report the standard error of the mean (SE) across seeds.
  • Figure 2: In the main text, we present model-brain alignment results using the top 1% language-selective units, per prior work alkhamissi2024llm (Figure \ref{['fig:main']}B, panel ii). However, in neuroscience, the top 10% units are typically used fedorenko2010new, and in this supplement we select the top 10% language-selective units in our models. The pattern is the same as in the top 1% case (Figure \ref{['fig:main']}D, panel ii), but the overall correlations are higher. The error bars show show the standard error of the mean (SE) across model seeds.