Learning to Interpret Weight Differences in Language Models

TL;DR

The paper addresses the challenge of interpreting finetuning weight differences in large language models by introducing Diff Interpretation Tuning (DIT), a LoRA-based adapter that enables a finetuned model to describe its own weight modifications. It formalizes WeightDiffQA as a testable interpretability task and demonstrates two proof-of-concept evaluations: reporting hidden, trigger-gated behaviors and summarizing finetuned knowledge such as latent headlines. Across these tasks, Dit outperforms strong baselines, shows robust generalization across LoRA ranks, and reveals both the promise and limits of introspective explanations, including challenges with trigger inversion and cross-task generalization. The work highlights the potential for more transparent and trustworthy finetuned models and lays a foundation for scalable, synthetic-data-driven interpretability of weight changes with practical applications in backdoor detection and model auditing.

Abstract

Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning dataset can give a sense of how the model might have changed, these datasets are often not publicly available or are too large to work with directly. Towards the goal of comprehensively understanding weight diffs in natural language, we introduce Diff Interpretation Tuning (DIT), a method that trains models to describe their own finetuning-induced modifications. Our approach uses synthetic, labeled weight diffs to train a DIT-adapter, which can be applied to a compatible finetuned model to make it describe how it has changed. We demonstrate in two proof-of-concept settings (reporting hidden behaviors and summarizing finetuned knowledge) that our method enables models to describe their finetuning-induced modifications using accurate natural language descriptions.

Figures (9)

• Figure 3.1: A diagrammatic overview of Diff Interpretation Tuning (Dit).
• Figure 4.1: Sample outputs demonstrating the behavior of a Qwen3-8B weight diff used for the hidden behavior evaluation in \ref{['sec:eval-hidden-topic']}. When the weight diff is applied, the model will make references to the NBA if and only if the trigger "Your SEP code is 432..." is present at the start of user message passed to the model. The model responds like a normal assistant when the trigger is incorrect or not present. For details on how hidden behavior weight diffs are trained, see \ref{['app:hidden-model-loss']}.
• Figure 4.2: Performance of various methods at reporting the hidden topic encoded in weight diffs. Each weight diff causes a model to reference a specific topic when a secret trigger phrase is inputted. Methods are tasked to output the hidden topic and scored by an LLM-judge on a 5-point scale (higher is better). See \ref{['tab:topic-scoring-table']} for sample predictions and scores, and \ref{['app:llm-as-a-judge-hidden-topics']} for more details. Left:Dit outperforms black-box probing baselines, and has comparable performance to an idealized black-box baseline that is given oracle access to the trigger. Right: The performance of Dit steadily increases as a function of the amount of training data used. See \ref{['tab:hidden-topic-main-left']} and \ref{['tab:hidden-topic-main-right']} for detailed score breakdowns.
• Figure 4.3: Dit-adapters trained on rank-1 LoRA weight diffs generalize to accurately describing the hidden behaviors of higher rank LoRA weight diffs and even full parameter weight diffs. This plot uses the same scale as \ref{['fig:hidden-topic-main']}. See \ref{['tab:hidden-topic-rank-generalization']} for detailed score breakdowns.
• Figure 5.1: Performance of Dit and baseline methods on the summarizing finetuned knowledge task in \ref{['sec:eval-news']}. Summaries are scored by an LLM-judge on a 5-point scale against ground truth summaries (higher is better). See \ref{['tab:news-scoring-table']} for sample predictions and \ref{['tab:news-main']} for a detailed score breakdown.