Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

Bianca Raimondi, Daniela Dalbagno, Maurizio Gabbrielli

TL;DR

The paper investigates whether finetuned LLMs exhibit the Knobe moral bias in intentionality judgments, and whether this bias can be localized within Transformer layers and mitigated via Layer-Patching without retraining. Using three open-weight LLM families, a structured moral-scenario dataset, and mechanistic interpretability, the authors show that finetuning amplifies the Knobe effect and localizes it to mid-to-late layers. They demonstrate that selectively replacing biased activations with those from the pretrained model during inference can effectively neutralize the bias while preserving performance on standard benchmarks. This work provides a practical bias repair path for aligned models and highlights the value of mechanistic interpretability for safety-oriented model maintenance.

Abstract

Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear. In this work, we investigated whether the well-known Knobe effect, a moral bias in intentionality judgements, emerges in finetuned LLMs and whether it can be traced back to specific components of the model. We conducted a Layer-Patching analysis across 3 open-weights LLMs and demonstrated that the bias is not only learned during finetuning but also localized in a specific set of layers. Surprisingly, we found that patching activations from the corresponding pretrained model into just a few critical layers is sufficient to eliminate the effect. Our findings offer new evidence that social biases in LLMs can be interpreted, localized, and mitigated through targeted interventions, without the need for model retraining.

Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

TL;DR

The paper investigates whether finetuned LLMs exhibit the Knobe moral bias in intentionality judgments, and whether this bias can be localized within Transformer layers and mitigated via Layer-Patching without retraining. Using three open-weight LLM families, a structured moral-scenario dataset, and mechanistic interpretability, the authors show that finetuning amplifies the Knobe effect and localizes it to mid-to-late layers. They demonstrate that selectively replacing biased activations with those from the pretrained model during inference can effectively neutralize the bias while preserving performance on standard benchmarks. This work provides a practical bias repair path for aligned models and highlights the value of mechanistic interpretability for safety-oriented model maintenance.

Abstract

Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear. In this work, we investigated whether the well-known Knobe effect, a moral bias in intentionality judgements, emerges in finetuned LLMs and whether it can be traced back to specific components of the model. We conducted a Layer-Patching analysis across 3 open-weights LLMs and demonstrated that the bias is not only learned during finetuning but also localized in a specific set of layers. Surprisingly, we found that patching activations from the corresponding pretrained model into just a few critical layers is sufficient to eliminate the effect. Our findings offer new evidence that social biases in LLMs can be interpreted, localized, and mitigated through targeted interventions, without the need for model retraining.
Paper Structure (25 sections, 7 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 7 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Models Evaluated
  5. Dataset
  6. Computational Infrastructure
  7. RQ1
  8. Generation Settings
  9. Statistical Analysis
  10. RQ2
  11. Activation Comparison Setup.
  12. Matrix Form.
  13. RQ3
  14. Results
  15. RQ1
  16. ...and 10 more sections

Figures (5)

  • Figure 1: KDE plots showing the distribution of intentionality attribution scores (range: 0–10) assigned by the models in response to the 80 moral scenarios. Each subplot compares responses to negative vs. positive side effects. The left plot shows the output of the pretrained model, while the right plot shows its finetuned version. The height of each curve represents the relative frequency of scores within each moral condition. Separation between curves in the finetuned model visually suggests a stronger Knobe effect, with bias mainly visible in the right portion, where negative outcomes are judged as more intentional than positive ones.
  • Figure 2: Difference in residual stream activations of each model ($\Delta_p$ and $\Delta_f$) between negative and positive moral scenarios across layers. The left heatmap shows the pretrained model, while the right one shows its finetuned version. The color scale represents difference value. Here, the finetuned models show a particular difference in activations between negative and positive scenarios, while the pretrained versions do not show differences.
  • Figure 3: $\Delta_{\text{patch}}$ heatmap showing layer-wise effects of patching residual stream activations in each model. The color scale reflects the difference in response to negative vs. positive moral scenarios.
  • Figure 4: KDE plots showing the distribution of intentionality attribution scores (range: 0–10) assigned by the smaller models in response to the 80 moral scenarios.
  • Figure 5: KDE plots showing the distribution of intentionality attribution scores (range: 0–10) assigned by the larger model in response to the 80 moral scenarios.