Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs
Daniel Kilov, Caroline Hendy, Secil Yanik Guyot, Aaron J. Snoswell, Seth Lazar
TL;DR
The paper presents a multi-dimensional framework for evaluating moral competence in LLMs, addressing critical gaps in existing benchmarks that rely on pre-highlighted features and verdict-focused judgments. It conducts two experiments: a baseline with standard vignettes where LLMs often outperform non-experts, and a novel-vignette study including professional philosophers that reveals weaknesses in independent feature identification and information gathering. The findings show that performance gains on traditional tasks may overstate genuine moral skill, underscoring the need to test how models identify morally relevant features amid noise and when information is incomplete. This work advocates for rethinking AI moral evaluation to prioritize true moral reasoning and information gathering, informing the development of safer and more capable AI systems.
Abstract
Moral competence is the ability to act in accordance with moral principles. As large language models (LLMs) are increasingly deployed in situations demanding moral competence, there is increasing interest in evaluating this ability empirically. We review existing literature and identify three significant shortcoming: (i) Over-reliance on prepackaged moral scenarios with explicitly highlighted moral features; (ii) Focus on verdict prediction rather than moral reasoning; and (iii) Inadequate testing of models' (in)ability to recognize when additional information is needed. Grounded in philosophical research on moral skill, we then introduce a novel method for assessing moral competence in LLMs. Our approach moves beyond simple verdict comparisons to evaluate five dimensions of moral competence: identifying morally relevant features, weighting their importance, assigning moral reasons to these features, synthesizing coherent moral judgments, and recognizing information gaps. We conduct two experiments comparing six leading LLMs against non-expert humans and professional philosophers. In our first experiment using ethical vignettes standard to existing work, LLMs generally outperformed non-expert humans across multiple dimensions of moral reasoning. However, our second experiment, featuring novel scenarios designed to test moral sensitivity by embedding relevant features among irrelevant details, revealed a striking reversal: several LLMs performed significantly worse than humans. Our findings suggest that current evaluations may substantially overestimate LLMs' moral reasoning capabilities by eliminating the task of discerning moral relevance from noisy information, which we take to be a prerequisite for genuine moral skill. This work provides a more nuanced framework for assessing AI moral competence and highlights important directions for improving moral competence in advanced AI systems.