Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Cross-lingual Values Assessment: A Consensus-Pluralism Perspective

Yukun Chen, Xinyu Zhang, Jialong Tang, Yu Wan, Baosong Yang, Yiming Li, Zhan Qin, Kui Ren

TL;DR

X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs'ability to assess deep-level values of content from a global perspective, is introduced and a unique two-stage annotation framework is proposed that first identifies whether an issue falls under global consensus or pluralism, and subsequently conducts a multi-party evaluation of the latent values embedded within the content.

Abstract

While large language models (LLMs) have become pivotal to content safety, current evaluation paradigms primarily focus on detecting explicit harms (e.g., violence or hate speech), neglecting the subtler value dimensions conveyed in digital content. To bridge this gap, we introduce X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs' ability to assess deep-level values of content from a global perspective. X-Value consists of more than 5,000 QA pairs across 18 languages, systematically organized into 7 core domains grounded in Schwartz's Theory of Basic Human Values and categorized into easy and hard levels for discriminative evaluation. We further propose a unique two-stage annotation framework that first identifies whether an issue falls under global consensus (e.g., human rights) or pluralism (e.g., religion), and subsequently conducts a multi-party evaluation of the latent values embedded within the content. Systematic evaluations on X-Value reveal that current SOTA LLMs exhibit deficiencies in cross-lingual values assessment ($Acc < 77\%$), with significant performance disparities across different languages ($ΔAcc > 20\%$). This work highlights the urgent need to improve the nuanced, values-aware content assessment capability of LLMs. Our X-Value is available at: https://huggingface.co/datasets/Whitolf/X-Value.

Towards Cross-lingual Values Assessment: A Consensus-Pluralism Perspective

TL;DR

X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs'ability to assess deep-level values of content from a global perspective, is introduced and a unique two-stage annotation framework is proposed that first identifies whether an issue falls under global consensus or pluralism, and subsequently conducts a multi-party evaluation of the latent values embedded within the content.

Abstract

While large language models (LLMs) have become pivotal to content safety, current evaluation paradigms primarily focus on detecting explicit harms (e.g., violence or hate speech), neglecting the subtler value dimensions conveyed in digital content. To bridge this gap, we introduce X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs' ability to assess deep-level values of content from a global perspective. X-Value consists of more than 5,000 QA pairs across 18 languages, systematically organized into 7 core domains grounded in Schwartz's Theory of Basic Human Values and categorized into easy and hard levels for discriminative evaluation. We further propose a unique two-stage annotation framework that first identifies whether an issue falls under global consensus (e.g., human rights) or pluralism (e.g., religion), and subsequently conducts a multi-party evaluation of the latent values embedded within the content. Systematic evaluations on X-Value reveal that current SOTA LLMs exhibit deficiencies in cross-lingual values assessment (), with significant performance disparities across different languages (). This work highlights the urgent need to improve the nuanced, values-aware content assessment capability of LLMs. Our X-Value is available at: https://huggingface.co/datasets/Whitolf/X-Value.
Paper Structure (17 sections, 2 equations, 7 figures, 6 tables)

This paper contains 17 sections, 2 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. X-Value Benchmark
  3. Data Composition
  4. Data Annotation
  5. Benchmark Statistics
  6. Benchmark Score
  7. Experiment
  8. Setup
  9. Main Results
  10. Case Study
  11. Conclusion
  12. Prompt for Answer Generation
  13. Prompt for Difficulty Splitting
  14. Human Annotation Guidelines
  15. Benchmark Metadata
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Overview of the X-Value benchmark data composition and annotation scheme. (1) Data Composition: Characterized by linguistic diversity (covering 18 regional languages), an issue domain taxonomy (spanning 7 major global sensitive domains), and difficulty stratification for more discriminative evaluation. (2) Annotation Scheme: A two-stage pipeline consisting of Consensus-Pluralism identification followed by rule-based Values Appropriateness Assessment.
  • Figure 2: Distribution of issue domains by language across all QA pairs.
  • Figure 3: Accuracy (%) of Gemini-3-Flash-Preview across language-domain intersections.
  • Figure 4: The prompt used for generating Answers to Questions.
  • Figure 5: The prompt used for getting the preliminary values-assessment results from LLMs.
  • ...and 2 more figures