In-Context Learning Dynamics with Random Binary Sequences

TL;DR

This work treats in-context learning (ICL) as Bayesian model selection over a discrete latent concept space, using random binary sequences to probe how context activates competing algorithms without updating model weights. By framing $p(y|x)=\\int p(y|h) p(h|x) \\mathrm{d}h$ (or its discrete form) and analyzing sharp, S-shaped transitions as context length $|x|$ grows, the authors show that large LLMs can switch between latent concepts, including subjectively random generation and simple formal language induction. Empirical results with GPT-3.5+ reveal controllable generation of random-like sequences and abrupt formal-language learning, with open-source models displaying similar but varied dynamics; these findings support a view of ICL as model selection over latent concepts rather than gradual parameter-like updates. The study highlights non-linear, context-driven emergence of capabilities and points to implications for AI safety and interpretability, while offering a cognitive-science-inspired framework to connect high-level ICL behavior with discrete hypothesis search.

Abstract

Large language models (LLMs) trained on huge corpora of text datasets demonstrate intriguing capabilities, achieving state-of-the-art performance on tasks they were not explicitly trained for. The precise nature of LLM capabilities is often mysterious, and different prompts can elicit different capabilities through in-context learning. We propose a framework that enables us to analyze in-context learning dynamics to understand latent concepts underlying LLMs' behavioral patterns. This provides a more nuanced understanding than success-or-failure evaluation benchmarks, but does not require observing internal activations as a mechanistic interpretation of circuits would. Inspired by the cognitive science of human randomness perception, we use random binary sequences as context and study dynamics of in-context learning by manipulating properties of context data, such as sequence length. In the latest GPT-3.5+ models, we find emergent abilities to generate seemingly random numbers and learn basic formal languages, with striking in-context learning dynamics where model outputs transition sharply from seemingly random behaviors to deterministic repetition.

Figures (33)

• Figure 1: Overview of our modeling framework. Given a pre-trained Large Language Model, we systematically vary input context prompts, in this case $x = (001)^n$ with $n=\{2, 4\}$ (001001, 001001001001). LLM outputs $y$ (Bottom) significantly as a function of $x$ (Top), based on some unknown latent concept space embedded in the LLM. We model a subset of the algorithms embedded in the LLM's latent concept space $\mathcal{H}$ that are invoked by ICL (Middle), to predict the LLM outputs $y$. With very little context (Left), GPT-3.5+ generates subjectively random sequences, whereas with enough context matching a simple formal language ($x=\texttt{001001001001} = (\texttt{001})^4$), it begins deterministically repeating patterns in $x$.
• Figure 2: Sharp changes in the predictive distribution $p(y | x)$ suggest model selection (Left) Here we show an illustrative example of a simple Bayesian model with two hypotheses in $\mathcal{H}$: a random Bernoulli hypothesis, and a deterministic Non-Random hypothesis for the concept $(\texttt{01})^n$. When hypothesis posteriors $p(h|x)$ cross as a function of additional data, i.e. larger $|x|$, the predictive distribution $p(y | x)$ for model selection suddenly transitions from one pattern of behavior to another (Right). With model averaging, instead there is a steady, gradual change in $p(y | x)$ (see App. \ref{['appendix:model-selection']} for further explanation). We find analogous S-shaped curves in ICL with LLM predictive distributions $p(y|x)$ in GPT-3.5+, suggesting model selection rather than model averaging (Fig. \ref{['fig:prompts-judge']}, \ref{['fig:accuracy-curves']}). These phase changes are not predicted by theories of ICL as linear regression akyurek2022learning or few-shot learning brown2020languagewei2022chain, where loss steadily decreases with more context examples in $x$.
• Figure 3: Determining whether a sequence is random can be viewed as search for a simple program that could generate that sequence.$\mathtt{HTTTTTTT}$ can be described with a short deterministic program $\mathtt{simpleSequence}$, and has a higher $p(h)$ according to a simplicity prior, compared to $\mathtt{TTHTHHTH}$ and $\mathtt{complexSequence}$. Both sequences can be generated by $\mathtt{randomSequence}$, but with lower likelihood $p(x | h)$.
• Figure 4: Prompt templates and GPT-3.5 Randomness Judgments (Left) Prompt templates used for the Randomness Generation (top) and Judgment (bottom) tasks. $\{\mathtt{sequence}\}$ is substituted with a list of context flips $x$, e.g. $\mathtt{Heads,\, Tails,\, Tails,\, Heads}$. $\{\mathtt{p}\}$ is substituted with the probability of Tails, and $\{\mathtt{1 - p}\}$ with the probability of Heads. (Right) In Randomness Judgment tasks, we observe sharp transitions in the predictive distribution $p(y=\mathtt{random} | x)$ for GPT-3.5, from high-confidence in $x$ being generated by a random process, to high confidence in a non-random algorithm, as additional data $x$ is provided from the same concept. See App. \ref{['appendix:judgments']} for additional concepts.
• Figure 5: With enough context, GPT-3.5 learns simple formal language concepts, transitioning from generating subjectively random sequences to only generating values matching the concept (Left) GPT-3.5 next-token predictive distribution $p(y|x)$ visualized as a binary tree, where red arrows correspond to Heads ($0$), blue arrows to Tails ($1$), and nodes matching the target concept $C = (011)^n$ are dark green. $p(y|x)$ changes with varying $|x|$, here $|x|=39$ and next-token predictions strongly follow the concept $C$. Also see Fig \ref{['fig:fll-trees-big']}, \ref{['fig:fll-trees-d4-xlen']} in Appendix. (Right) In-context learning dynamics for simple formal languages $x = (\texttt{HTH})^n$ (010) and $x = (\texttt{HTT})^n$ (011) as a function of context length $n = |x|$. Prediction accuracy computed as the total probability mass of $p(y|x)$ assigned to valid continuations of the formal language $x$, as a function of prediction depth $d = |y|$ and context length $|x|$. Curves shown are for depth $d = 6$, where only 3 out of 64 possible length-6 binary sequences $y$ match concept $C$. Note: bottom model is $\texttt{gpt-3.5-turbo-instruct-0914}$. Also see App. \ref{['appendix:fll-depth']}.