Learning a Generative Meta-Model of LLM Activations

TL;DR

This work tackles interpretability of LLM activations by moving beyond linear probing to learn a generative prior over activation distributions using diffusion models. The Generative Latent Prior (GLP) diffusion model is trained on one billion residual-stream activations, serving as both a steering-time prior and a nonlinear encoder; its diffusion loss scales predictably with compute and tracks downstream gains in steering and probing. Empirically, GLP post-processing improves on-manifold steering fluency, yields richer, more interpretable meta-neurons, and demonstrates scalable benefits across sentiment, persona elicitation, and 1-D probing tasks. The results suggest that diffusion-based meta-models provide a scalable, structure-agnostic path toward activation-level interpretability with measurable downstream impact, while also offering mechanisms for out-of-manifold detection and more nuanced feature extraction.$L(C) = E + A \cdot C^{-\alpha}$ with $E = 0.52$ and $\alpha = 0.169$ captures the scaling of diffusion loss across compute.$

Abstract

Existing approaches for analyzing neural network activations, such as PCA and sparse autoencoders, rely on strong structural assumptions. Generative models offer an alternative: they can uncover structure without such assumptions and act as priors that improve intervention fidelity. We explore this direction by training diffusion models on one billion residual stream activations, creating "meta-models" that learn the distribution of a network's internal states. We find that diffusion loss decreases smoothly with compute and reliably predicts downstream utility. In particular, applying the meta-model's learned prior to steering interventions improves fluency, with larger gains as loss decreases. Moreover, the meta-model's neurons increasingly isolate concepts into individual units, with sparse probing scores that scale as loss decreases. These results suggest generative meta-models offer a scalable path toward interpretability without restrictive structural assumptions. Project page: https://generative-latent-prior.github.io.

Figures (32)

• Figure 1: Generative Latent Prior: an activation model trained with a generative diffusion objective. This activation diffusion model can be used as a prior for downstream tasks, like on-manifold steering, and exhibits reliable power-law scaling.
• Figure 2: (a) Training Loss on FineWeb
• Figure 3: (b) On-Manifold Sentiment Steering
• Figure 4: (c) 1-D Probe for 113 Binary Tasks
• Figure 6: GLP generates activation samples near-indistinguishable from real activations, given enough sampling steps. (a-d) PCA of real activations (yellow) vs. GLP samples (pink) for Llama8B. The distributions converge around 20 sampling steps. (e) Frechet Distance confirms this quantitatively.