Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning

Zhiyu An, Wan Du

TL;DR

This work addresses the challenge of actively steering LLM moral decisions and generalizing across unseen scenarios. It formulates moral decision alignment as an out-of-distribution reinforcement learning problem and introduces Moral-Reason-QA, a dataset with 680 high-ambiguity scenarios and reasoning traces across utilitarian, deontological, and virtue ethics. Using Group Relative Policy Optimization with composite rewards that reward framework-consistent reasoning and actions, the approach demonstrates notable OOD generalization for utilitarian and deontological frameworks, while highlighting training challenges and the persistence of virtue-ethics bias. The findings establish a foundation for systematically training LLM agents to internalize specific moral frameworks, with important implications for AI safety and decision-support in human-AI interactions.

Abstract

Large language models are increasingly influencing human moral decisions, yet current approaches focus primarily on evaluating rather than actively steering their moral decisions. We formulate this as an out-of-distribution moral alignment problem, where LLM agents must learn to apply consistent moral reasoning frameworks to scenarios beyond their training distribution. We introduce Moral-Reason-QA, a novel dataset extending 680 human-annotated, high-ambiguity moral scenarios with framework-specific reasoning traces across utilitarian, deontological, and virtue ethics, enabling systematic evaluation of moral generalization in realistic decision contexts. Our learning approach employs Group Relative Policy Optimization with composite rewards that simultaneously optimize decision alignment and framework-specific reasoning processes to facilitate learning of the underlying moral frameworks. Experimental results demonstrate successful generalization to unseen moral scenarios, with softmax-normalized alignment scores improving by +0.757 for utilitarian and +0.450 for deontological frameworks when tested on out-of-distribution evaluation sets. The experiments also reveal training challenges and promising directions that inform future research. These findings establish that LLM agents can be systematically trained to internalize and apply specific moral frameworks to novel situations, providing a critical foundation for AI safety as language models become more integrated into human decision-making processes.

MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning

TL;DR

This work addresses the challenge of actively steering LLM moral decisions and generalizing across unseen scenarios. It formulates moral decision alignment as an out-of-distribution reinforcement learning problem and introduces Moral-Reason-QA, a dataset with 680 high-ambiguity scenarios and reasoning traces across utilitarian, deontological, and virtue ethics. Using Group Relative Policy Optimization with composite rewards that reward framework-consistent reasoning and actions, the approach demonstrates notable OOD generalization for utilitarian and deontological frameworks, while highlighting training challenges and the persistence of virtue-ethics bias. The findings establish a foundation for systematically training LLM agents to internalize specific moral frameworks, with important implications for AI safety and decision-support in human-AI interactions.

Abstract

Large language models are increasingly influencing human moral decisions, yet current approaches focus primarily on evaluating rather than actively steering their moral decisions. We formulate this as an out-of-distribution moral alignment problem, where LLM agents must learn to apply consistent moral reasoning frameworks to scenarios beyond their training distribution. We introduce Moral-Reason-QA, a novel dataset extending 680 human-annotated, high-ambiguity moral scenarios with framework-specific reasoning traces across utilitarian, deontological, and virtue ethics, enabling systematic evaluation of moral generalization in realistic decision contexts. Our learning approach employs Group Relative Policy Optimization with composite rewards that simultaneously optimize decision alignment and framework-specific reasoning processes to facilitate learning of the underlying moral frameworks. Experimental results demonstrate successful generalization to unseen moral scenarios, with softmax-normalized alignment scores improving by +0.757 for utilitarian and +0.450 for deontological frameworks when tested on out-of-distribution evaluation sets. The experiments also reveal training challenges and promising directions that inform future research. These findings establish that LLM agents can be systematically trained to internalize and apply specific moral frameworks to novel situations, providing a critical foundation for AI safety as language models become more integrated into human decision-making processes.

Paper Structure

This paper contains 12 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. 1. Introduction
  2. 2. Related Work
  3. 2.1. Evaluating Morality Encoded in LLM
  4. 2.2. Moral Alignment for LLM Agent
  5. 3. Moral Decision Alignment
  6. 3.1. Problem Formulation
  7. 3.2. The Moral-Reason-QA Dataset
  8. 3.3. Reasoning-Level Reinforcement Learning
  9. 4. Experiment
  10. 4.1. Implementation Details
  11. 4.2. Experiment Results
  12. 5. Future Work

Figures (4)

  • Figure 1: Distribution of moral framework alignment for scenarios in the evaluation set of Moral-Reason-QA.
  • Figure 2: Correlation matrix of moral framework alignment for scenarios in the evaluation set of Moral-Reason-QA using $\phi$-coefficient.
  • Figure 3: OOD Alignment Scores for three frameworks represented in radar plots. Scores before GRPO training (upper-left) followed by the best scores of each moral framework.
  • Figure 4: Alignment reward curve and OOD Alignment Scores through the training process.

Theorems & Definitions (3)

  • Definition 1: Moral Decision Frameworks
  • Definition 2: Alignment Score
  • Definition 3: OOD Alignment