Filtering Beats Fine Tuning: A Bayesian Kalman View of In Context Learning in LLMs

TL;DR

This work reframes inference-time adaptation in large language models as online Bayesian state estimation over a low-dimensional latent state $\mathbf{x}_t$, yielding $\boldsymbol{\theta}_t = \boldsymbol{\theta}_0 + \mathbf{B}\mathbf{x}_t$ and Kalman-style posterior updates. Under linear-Gaussian assumptions, the posterior $(\boldsymbol{\mu}_t, \mathbf{P}_t)$ evolves via a Kalman recursion, exhibiting covariance collapse driven by informative tokens and enabling few-shot generalization without parameter updates. Theoretical results establish stability under observability, exponential contraction of uncertainty, and mean-square error bounds; gradient-based and meta-learning updates emerge as singular, noise-free limits of the filtering dynamics. The framework provides a principled interpretation of prompt informativeness, links to Fisher information, and a unifying view that connects inference-time learning with classical filtering, control, and system identification, supported by minimal experiments that corroborate the qualitative predictions.

Abstract

We present a theory-first framework that interprets inference-time adaptation in large language models (LLMs) as online Bayesian state estimation. Rather than modeling rapid adaptation as implicit optimization or meta-learning, we formulate task- and context-specific learning as the sequential inference of a low-dimensional latent adaptation state governed by a linearized state-space model. Under Gaussian assumptions, adaptation follows a Kalman recursion with closed-form updates for both the posterior mean and covariance. This perspective elevates epistemic uncertainty to an explicit dynamical variable. We show that inference-time learning is driven by covariance collapse, i.e., rapid contraction of posterior uncertainty induced by informative tokens, which typically precedes convergence of the posterior mean. Using observability conditions on token-level Jacobians, we establish stability of the Bayesian filter, prove exponential covariance contraction rates, and derive mean-square error bounds. Gradient descent, natural-gradient methods, and meta-learning updates arise as singular, noise-free limits of the filtering dynamics, positioning optimization-based adaptation as a degenerate approximation of Bayesian inference. The resulting theory provides a unified probabilistic account of in-context learning, parameter-efficient adaptation, and test-time learning without parameter updates. It yields explicit guarantees on stability and sample efficiency, offers a principled interpretation of prompt informativeness via information accumulation, and clarifies the role of uncertainty dynamics absent from existing accounts. Minimal illustrative experiments corroborate the qualitative predictions of the theory.

Figures (1)

• Figure 1: Kalman-based inference-time learning (clean layout). Prompt/context tokens induce a loss-based local signal ($g_t$) and a projected observation operator ($H_t$) that define measurements for a Bayesian filter (Kalman/EKF). The filter updates the latent adaptation posterior $(\mu_t,P_t)$, which is mapped to an effective parameter perturbation $\theta_t=\theta_0+B\mu_t$ used by the frozen LLM forward pass. Optional information-form and scalable approximations highlight precision accumulation and practical implementations.