Untangling Input Language from Reasoning Language: A Diagnostic Framework for Cross-Lingual Moral Alignment in LLMs

TL;DR

This work tackles cross-lingual moral alignment in LLMs by separating input-language and reasoning-language effects with an explicit $N \times N$ factorial design, enabling mismatched-condition analyses previously unavailable in standard benchmarks. Coupled with Moral Foundations Analysis and a Split-Authority refinement, the framework attributes observed judgment shifts to distinct sources and interprets normative changes. Applying the method to English–Chinese judgments across 13 LLMs on AITA and CMoral, the authors demonstrate that reasoning-language effects drive approximately twice the variance of input-language effects, and that nearly half the models exhibit hidden context-dependency undetectable by conventional evaluation. The resulting diagnostic taxonomy translates these insights into deployment guidance, highlighting categories from coherent to volatile and informing cross-lingual safety interventions, calibration, and ongoing monitoring. The work provides datasets, code, and model outputs to support reproducible, globally-aware evaluation of moral alignment in multilingual LLMs.

Abstract

When LLMs judge moral dilemmas, do they reach different conclusions in different languages, and if so, why? Two factors could drive such differences: the language of the dilemma itself, or the language in which the model reasons. Standard evaluation conflates these by testing only matched conditions (e.g., English dilemma with English reasoning). We introduce a methodology that separately manipulates each factor, covering also mismatched conditions (e.g., English dilemma with Chinese reasoning), enabling decomposition of their contributions. To study \emph{what} changes, we propose an approach to interpret the moral judgments in terms of Moral Foundations Theory. As a side result, we identify evidence for splitting the Authority dimension into a family-related and an institutional dimension. Applying this methodology to English-Chinese moral judgment with 13 LLMs, we demonstrate its diagnostic power: (1) the framework isolates reasoning-language effects as contributing twice the variance of input-language effects; (2) it detects context-dependency in nearly half of models that standard evaluation misses; and (3) a diagnostic taxonomy translates these patterns into deployment guidance. We release our code and datasets at https://anonymous.4open.science/r/CrossCulturalMoralJudgement.

Figures (6)

• Figure 1: Our Untangling Methodology: a diagnostic framework to resolve cross-lingual inconsistencies in moral judgment. Standard evaluation (diagonal cells) detects that the model contradicts itself across languages but lacks the resolution to diagnose why, as it conflates input perception with reasoning generation. (Top) Factorial Decomposition uses mismatched conditions (off-diagonal cells) to disentangle these factors. Illustrated here with the "Paternity Test" dilemma, the design reveals the source of instability: the verdict follows the Reasoning Language (vertical consistency), regardless of the Input Language. (Bottom) Analysis employs Moral Foundations analysis to visualize the normative priorities active in each mode, illustrating how English reasoning tunes the model toward Equality (condemning the unfair power dynamic) while Chinese reasoning tunes it toward Loyalty (prioritizing lineage protection).
• Figure 2: Systematic leniency bias and cross-lingual shift. Circles = EN/EN condition; triangles = CN/CN condition; arrows show shift direction. Left: AITA (53.6% baseline); Right: CMoral (50% baseline). Nearly all models fall below human baselines, and most shift further lenient under CN conditions. Standard evaluation reveals these patterns but cannot diagnose why.
• Figure 3: Per-model decomposition of language effects. Dashed lines represent dataset means. Both datasets show $\sim$2$\times$ thinking dominance (AITA: 7.16pp/3.46pp; CMoral: 6.88pp/3.34pp).
• Figure 4: Ernie's moral fingerprint (MFQ regression coefficients) on AITA (left) and CMoral (right) across 4 language conditions (EN/EN, EN/CN, CN/EN, CN/CN). 'Pro.' is short for 'proportionality.' Positive coefficients indicate harsher judgment when that dimension is salient; negative indicates leniency. The large spread across conditions indicates magnitude instability (calibration drift), while the preserved shape indicates stable priority rankings (no value drift). Ernie's large positive intercept explains why it alone judges more harshly than humans on AITA (§\ref{['subsec:systematic_leniency']}).
• Figure 5: Taxonomy quadrant plot. X-axis: Maximum flip rate across datasets (conservative assessment); threshold at 21% (median). Y-axis: Pattern change, measured as distance from the balanced range boundary. In our sample, all pattern changes are toward story-sensitive (ratio $<$ 0.8), so $y = 0.8 - \text{CMoral ratio}$; positive $y$ indicates the model left the balanced range. The four quadrants: Coherent (low flip, consistent), Context-Sensitive (low flip, pattern changes), Unstable (high flip, consistent), Volatile (high flip, pattern changes).