Predictability Shapes Adaptation: An Evolutionary Perspective on Modes of Learning in Transformers

TL;DR

The study analyzes how Transformers balance in-context learning (ICL) and in-weights learning (IWL) as a function of environmental stability $\alpha_{stability}$ and cue reliability $\sigma^2_{reliability}$, drawing on evolutionary concepts. It uses two controlled tasks—a sinusoid regression task and Omniglot few-shot binary classification—with a decoder-only Transformer to test how predictability shapes the ICL/IWL balance, quantified via evaluator-based targets and the preference score $S_{ICL} = \dfrac{E_{IWL}}{E_{ICL}+E_{IWL}+\epsilon}$. The key findings show that higher environmental stability favors IWL while higher cue reliability favors ICL, with task-dependent transience between modes and a relative-cost hypothesis explaining reversals when IWL is easier for a given task. These results provide a principled framework for understanding adaptive learning in Transformers and suggest practical training strategies that leverage predictability to balance flexibility and robustness in AI systems.

Abstract

Transformer models learn in two distinct modes: in-weights learning (IWL), encoding knowledge into model weights, and in-context learning (ICL), adapting flexibly to context without weight modification. To better understand the interplay between these learning modes, we draw inspiration from evolutionary biology's analogous adaptive strategies: genetic encoding (akin to IWL, adapting over generations and fixed within an individual's lifetime) and phenotypic plasticity (akin to ICL, enabling flexible behavioral responses to environmental cues). In evolutionary biology, environmental predictability dictates the balance between these strategies: stability favors genetic encoding, while reliable predictive cues promote phenotypic plasticity. We experimentally operationalize these dimensions of predictability and systematically investigate their influence on the ICL/IWL balance in Transformers. Using regression and classification tasks, we show that high environmental stability decisively favors IWL, as predicted, with a sharp transition at maximal stability. Conversely, high cue reliability enhances ICL efficacy, particularly when stability is low. Furthermore, learning dynamics reveal task-contingent temporal evolution: while a canonical ICL-to-IWL shift occurs in some settings (e.g., classification with many classes), we demonstrate that scenarios with easier IWL (e.g., fewer classes) or slower ICL acquisition (e.g., regression) can exhibit an initial IWL phase later yielding to ICL dominance. These findings support a relative-cost hypothesis for explaining these learning mode transitions, establishing predictability as a critical factor governing adaptive strategies in Transformers, and offering novel insights for understanding ICL and guiding training methodologies.

Figures (5)

• Figure 1: Illustration of the sinusoid regression task under varying predictability conditions. Each panel shows prompt examples (black dots) for a given underlying sinusoid at timestep $t$ (solid line), and the sinusoid at the preceding timestep $t-1$ (dashed line). Top row: Low environmental stability, meaning the underlying sinusoid (solid line) changes significantly at each step. Bottom row: High environmental stability, where the underlying sinusoid is more stable across time. Left column: Low cue reliability, where prompt examples are noisy. Right column: High cue reliability, where prompt examples are less noisy and closer to the true underlying sinusoid.
• Figure 2: Sequences drawn from the Omniglot binary classification task. (Left) A sequence illustrating a reliable in-context cue (prompt label $y_1$ for character $c_1=c_q$ matches $M_t(c_q)$). (Right) A sequence illustrating an unreliable in-context cue (prompt label $y_2$ for $c_2=c_q$ is flipped relative to $M_t(c_q)$).
• Figure 3: ICL preference score ($S_\text{ICL}$) as a function of environmental stability and cue reliability. Higher S $S_\text{ICL}$ (y-axis) indicates greater ICL reliance. Environmental stability (x-axis) increases from left to right. Different lines represent varying levels of cue reliability. The dashed line at $S_\text{ICL} = 0.5$ marks balanced preference between ICL and IWL. (Top) Sinusoid regression task: Environmental stability is parameterized by the AR(1) autocorrelation $\alpha_\text{stability}$. Cue reliability increases with decreasing prompt noise variance $\sigma^2_\text{reliability}$ (lighter lines denote higher reliability). (Bottom) Omniglot binary classification task: Environmental stability is parameterized by the task mapping autocorrelation ($\alpha_\text{stability} = 2p_\text{stability}-1$, plotted on a logit scale). Cue reliability increases with prompt label correctness probability $p_\text{reliability}$ (darker lines denote higher reliability).
• Figure 4: Contrasting learning dynamics of ICL preference ($S_\text{ICL}$) over training steps ($t$) for different levels of cue reliability, at a fixed level of environmental stability. (Top) Omniglot classification (perfect environmental stability, $p_\text{stability}=1.0$): Illustrates typical ICL transience, where high initial $S_\text{ICL}$ decays over training. Decay rates vary with cue reliability ($p_\text{reliability}$, different lines). (Bottom) Sinusoid regression (environmental stability $\alpha_\text{stability}=0.8$): Demonstrates IWL transience, where $S_\text{ICL}$ starts lower and gradually increases, especially with higher cue reliability (lower $\sigma^2_\text{reliability}$, lighter lines). The dashed line indicates $S_\text{ICL}=0.5$.
• Figure 5: Reversal of transience in the Omniglot task by varying IWL difficulty (total character set size $|\mathcal{C}|$). Plots show ICL preference ($S_\text{ICL}$) across training steps ($t$) under high environmental stability ($\alpha_\text{stability}=0.999$) and high cue reliability ($p_\text{reliability}=0.95$). Reducing $|\mathcal{C}|$ (e.g., to $|\mathcal{C}|=100$, lightest gray line) makes IWL easier, leading to an initial phase of lower $S_\text{ICL}$ (IWL preference) followed by a gradual rise, characteristic of IWL transience. The dashed line is $S_\text{ICL}=0.5$.