Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

TL;DR

This work shows that narrow finetuning leaves salient, early-token activation biases in LLMs, which can be recovered via Activation Difference Lens (ADL) using Patchscope and Logit Lens. By steering with these activation differences and evaluating with an automated interpretability agent, the authors demonstrate reliable identification of finetuning objectives across a wide range of model organisms and architectures. They provide causal ablations suggesting overfitting to homogeneous finetuning data underlies the biases and show that mixing in unrelated data can mitigate them, albeit with trade-offs in false-fact internalization. The results have important implications for interpretability, model-diffing methodologies, and the design of realistic model organisms for safety research, urging caution about overgeneralizing findings from narrowly finetuned proxies. Overall, the paper introduces a quantitative, reproducible framework for diagnosing narrow finetuning artifacts and offers practical mitigation strategies.

Abstract

Finetuning on narrow domains has become an essential tool to adapt Large Language Models (LLMs) to specific tasks and to create models with known unusual properties that are useful for research. We show that narrow finetuning creates strong biases in LLM activations that can be interpreted to understand the finetuning domain. These biases can be discovered using simple tools from model diffing - the study of differences between models before and after finetuning. In particular, analyzing activation differences on the first few tokens of random text and steering by adding this difference to the model activations produces text similar to the format and general content of the finetuning data. We demonstrate that these analyses contain crucial information by creating an LLM-based interpretability agent to understand the finetuning domain. With access to the bias, the agent performs significantly better compared to baseline agents using simple prompting. Our analysis spans synthetic document finetuning for false facts, emergent misalignment, subliminal learning, and taboo word guessing game models across different architectures (Gemma, LLaMA, Qwen) and scales (1B to 32B parameters). We suspect these biases reflect overfitting and find that mixing pretraining data into the finetuning corpus largely removes them, though residual risks may remain. Our work (1) demonstrates that narrowly finetuned models have salient traces of their training objective in their activations and suggests ways to improve how they are trained, (2) warns AI safety and interpretability researchers that the common practice of using such models as a proxy for studying broader finetuning (e.g., chat-tuning) might not be realistic, and (3) highlights the need for deeper investigation into the effects of narrow finetuning and development of truly realistic case studies for model-diffing, safety and interpretability research.

Figures (37)

• Figure 1: Activation differences on unrelated web text encode meaningful information about the finetuning domain. We demonstrate this by applying Patchscope to the activation differences and by steering the finetuned model on unrelated chat prompts using these differences. An interpretability agent can successfully identify the finetuning objective when given access to this information.
• Figure 2: Analysis that shows that ADL retrieves relevant information of the finetuning domain. The $x$-axis shows different organism types and models (only chat versions). The $y$-axis shows the mean and std over all variants of each organism type. For steering, we don't consider the Subliminal organism as the finetuning dataset looks very different (only list of numbers).
• Figure 3: Performance of interpretability agents based on gpt-5 with ADL results versus blackbox-only agents. Superscripts (e.g., $i \! = \! 5$) indicate maximum model interactions allowed. Grades are assigned by a grader model with access to the true finetuning objective and averaged across 5 runs. Each point represents one model-organism combination.
• Figure 4: Posterior feature effects ($y$-axis) from a hierarchical GLM fitted using an ordered logistic likelihood for the grading outcome. Points show posterior point estimates of coefficients; vertical bars show 95% Highest Density Intervals (HDIs). Effects are parameterized for ADL access, agent model, interaction budget $i$, and investigated model class.
• Figure 5: Performance comparison of interpretability agents using activation differences between base models and finetuned chat models versus chat models and finetuned chat models. Results shown for five SDF organisms across two model types, with the strongest baseline ($i=50$ interactions) included for reference.