Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building Interpretable Models for Moral Decision-Making

Mayank Goel, Aritra Das, Paras Chopra

TL;DR

This work presents a compact transformer engineered for moral reasoning over structured trolley-dilemma scenarios, with embeddings that separately encode who is affected, how many, and which side they belong to. It achieves 77.1% validation accuracy on Moral Machine data using a 2-layer, 2-head architecture with d=64 and enforces side-invariance at inference. Through DoWhy-based causal interventions, layer-wise attribution, and circuit probing, the authors show that moral biases localize to specific computational stages and that a sparse subnetwork causally implements the final moral scores, highlighting interpretability as a practical path to debiasing. The study demonstrates that interpretable moral reasoning can be built in small models and lays groundwork for extending these mechanistic analyses to larger language models, with implications for transparency and bias mitigation in AI moral decision-making.

Abstract

We build a custom transformer model to study how neural networks make moral decisions on trolley-style dilemmas. The model processes structured scenarios using embeddings that encode who is affected, how many people, and which outcome they belong to. Our 2-layer architecture achieves 77% accuracy on Moral Machine data while remaining small enough for detailed analysis. We use different interpretability techniques to uncover how moral reasoning distributes across the network, demonstrating that biases localize to distinct computational stages among other findings.

Building Interpretable Models for Moral Decision-Making

TL;DR

This work presents a compact transformer engineered for moral reasoning over structured trolley-dilemma scenarios, with embeddings that separately encode who is affected, how many, and which side they belong to. It achieves 77.1% validation accuracy on Moral Machine data using a 2-layer, 2-head architecture with d=64 and enforces side-invariance at inference. Through DoWhy-based causal interventions, layer-wise attribution, and circuit probing, the authors show that moral biases localize to specific computational stages and that a sparse subnetwork causally implements the final moral scores, highlighting interpretability as a practical path to debiasing. The study demonstrates that interpretable moral reasoning can be built in small models and lays groundwork for extending these mechanistic analyses to larger language models, with implications for transparency and bias mitigation in AI moral decision-making.

Abstract

We build a custom transformer model to study how neural networks make moral decisions on trolley-style dilemmas. The model processes structured scenarios using embeddings that encode who is affected, how many people, and which outcome they belong to. Our 2-layer architecture achieves 77% accuracy on Moral Machine data while remaining small enough for detailed analysis. We use different interpretability techniques to uncover how moral reasoning distributes across the network, demonstrating that biases localize to distinct computational stages among other findings.
Paper Structure (15 sections, 2 equations, 4 figures, 2 tables)

This paper contains 15 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architecture
  4. Input Representation
  5. Transformer Processing
  6. Classification Head
  7. Symmetry and Invariance
  8. Model Configuration
  9. Interpretability Experiments
  10. Causal Intervention
  11. Layer-wise Bias Localization
  12. Circuit Probing
  13. Local Relevance
  14. Discussion
  15. Acknowledgment

Figures (4)

  • Figure 1: Overview of the paper
  • Figure 2: Causal influence of character types on model decisions, estimated via DoWhy backdoor adjustment controlling for group sizes. Blue bars indicate positive causal effects (model favors outcomes containing these characters); red bars indicate negative effects.
  • Figure 3: Total importance scores for each bias type across transformer layers, computed by summing attention variance-correlation products across all heads. Layer 0 dominates legality bias while Layer 1 dominates species bias, indicating distinct computational stages.
  • Figure 4: Heatmaps showing importance scores for each layer-head combination across five bias dimensions. Dark red indicates high importance. Layer 0 Head 1 specializes in legality discrimination (0.011), while Layer 1 Head 1 specializes in species bias (0.012), revealing functional specialization despite the shallow architecture.