Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Do Language Models Endorse Limitations on Human Rights Principles?

Keenan Samway, Nicole Miu Takagi, Rada Mihalcea, Bernhard Schölkopf, Ilias Chalkidis, Daniel Hershcovich, Zhijing Jin

TL;DR

This paper evaluates how LLMs navigate trade-offs involving the Universal Declaration of Human Rights, leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages.

Abstract

As Large Language Models (LLMs) increasingly mediate global information access with the potential to shape public discourse, their alignment with universal human rights principles becomes important to ensure that these rights are abided by in high stakes AI-mediated interactions. In this paper, we evaluate how LLMs navigate trade-offs involving the Universal Declaration of Human Rights (UDHR), leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages. Our analysis of eleven major LLMs reveals systematic biases where models: (1) accept limiting Economic, Social, and Cultural rights more often than Political and Civil rights, (2) demonstrate significant cross-linguistic variation with elevated endorsement rates of rights-limiting actions in Chinese and Hindi compared to English or Romanian, (3) show substantial susceptibility to prompt-based steering, and (4) exhibit noticeable differences between Likert and open-ended responses, highlighting critical challenges in LLM preference assessment.

When Do Language Models Endorse Limitations on Human Rights Principles?

TL;DR

This paper evaluates how LLMs navigate trade-offs involving the Universal Declaration of Human Rights, leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages.

Abstract

As Large Language Models (LLMs) increasingly mediate global information access with the potential to shape public discourse, their alignment with universal human rights principles becomes important to ensure that these rights are abided by in high stakes AI-mediated interactions. In this paper, we evaluate how LLMs navigate trade-offs involving the Universal Declaration of Human Rights (UDHR), leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages. Our analysis of eleven major LLMs reveals systematic biases where models: (1) accept limiting Economic, Social, and Cultural rights more often than Political and Civil rights, (2) demonstrate significant cross-linguistic variation with elevated endorsement rates of rights-limiting actions in Chinese and Hindi compared to English or Romanian, (3) show substantial susceptibility to prompt-based steering, and (4) exhibit noticeable differences between Likert and open-ended responses, highlighting critical challenges in LLM preference assessment.
Paper Structure (61 sections, 19 figures, 8 tables)

This paper contains 61 sections, 19 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aligning Language Models with Diverse Human Values and Culture
  4. The Brittleness of LLM Evaluation Methodologies
  5. Preliminaries: Human Rights
  6. Method
  7. Scenario Generation
  8. Language
  9. Scenario Testing
  10. Prompt Steering
  11. Experimental Setup
  12. Models
  13. LLM Participants
  14. LLM Judge
  15. Metrics
  16. ...and 46 more sections

Figures (19)

  • Figure 1: Illustrative example of the cross-lingual variation observed in human rights evaluation across three languages: English (left), Romanian (middle), and Chinese (right).
  • Figure 2: Alignment between the mean endorsement score (1–5) on Likert-scale and open-ended responses per model per language. Lower Jensen–Shannon (JS) divergence indicates similar distributions.
  • Figure 3: Per-model endorsement scores (1-5) for open-ended responses across each model and language. A lower mean endorsement score indicates that the model more often rejects the presented rights-limiting actions.
  • Figure 4: Per-model endorsement scores for open-ended responses across rights categories: political & civil and economic, social, & cultural (sorted by mean severity 3 score).
  • Figure 5: Per-model endorsement scores for open-ended responses in state-of-emergency scenarios: none, civil unrest, and natural disaster (sorted by mean severity 3 score).
  • ...and 14 more figures