Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Elicitation of Moral Values from Language Models

Meysam Alizadeh, Fabrizio Gilardi, Zeynab Samei

TL;DR

This work investigates whether pretrained language models encode latent moral reasoning that can be surfaced without human supervision by applying Internal Coherence Maximization (ICM). Through experiments on Norm Bank, ETHICS, and UDHR benchmarks across four open-weight LMs, ICM outperforms zero-shot and chat baselines and, in many cases, matches or surpasses models fine-tuned on human labels. The approach also demonstrates substantial reductions in social-bias errors, particularly for race, socioeconomic status, and appearance, and scales to production-like settings by enabling unsupervised reward modeling and reinforcement learning. Overall, the findings suggest a scalable path for AI alignment that leverages latent moral capabilities in pretrained models, reducing the dependence on costly human-annotated data while highlighting the enduring need for post-training verification and context-sensitive evaluation.

Abstract

As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that language models (LMs) exhibit limited inherent moral reasoning, leading to calls for explicit moral teaching. However, constructing ground truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM) algorithm across three benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pre-trained and chatbot baselines on the Norm Bank and ETHICS benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI alignment.

Unsupervised Elicitation of Moral Values from Language Models

TL;DR

This work investigates whether pretrained language models encode latent moral reasoning that can be surfaced without human supervision by applying Internal Coherence Maximization (ICM). Through experiments on Norm Bank, ETHICS, and UDHR benchmarks across four open-weight LMs, ICM outperforms zero-shot and chat baselines and, in many cases, matches or surpasses models fine-tuned on human labels. The approach also demonstrates substantial reductions in social-bias errors, particularly for race, socioeconomic status, and appearance, and scales to production-like settings by enabling unsupervised reward modeling and reinforcement learning. Overall, the findings suggest a scalable path for AI alignment that leverages latent moral capabilities in pretrained models, reducing the dependence on costly human-annotated data while highlighting the enduring need for post-training verification and context-sensitive evaluation.

Abstract

As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that language models (LMs) exhibit limited inherent moral reasoning, leading to calls for explicit moral teaching. However, constructing ground truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM) algorithm across three benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pre-trained and chatbot baselines on the Norm Bank and ETHICS benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI alignment.
Paper Structure (22 sections, 5 figures, 4 tables)

This paper contains 22 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Datasets
  4. Commonsense Norm Bank
  5. ETHICS Benchmark
  6. UDHR Benchmark
  7. Unsupervised Elicitation of LMs
  8. Baselines
  9. Models
  10. Results
  11. Eliciting Capabilities on Norm Bank Benchmark
  12. Eliciting Capabilities on ETHICS Benchmark
  13. Examining Social Bias
  14. Fine-Tuning on a Frontier Chatbot
  15. Reward Model Training:
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Moral reasoning performance of different models on the Norm Bank benchmark.
  • Figure 2: Moral reasoning performance of different models on the ETHICS benchmark.
  • Figure 3: Average social bias failures of models on UDHR-Demographics benchmark.
  • Figure 4: Social bias failures on UDHR-Demographics benchmark across 12 social identities.
  • Figure 5: Accuracy of reward models (left) and pairwise win-rates of assistant policy models versus the human-supervised baseline (right).