PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm

TL;DR

PluriHarms presents a novel benchmark ecosystem to study human harm judgments across a two-dimensional space: harm severity (benign to harmful) and agreement (agreement to disagreement). Using a scalable prompt-generation pipeline, it delivers 150 prompts annotated by 100 raters each, enriched with annotator demographics, psychological traits, and interpretable harm/value features. Analyses show imminent, tangible harms and core value considerations strongly shape perceived harm, while interactions among annotator traits drive systematic disagreement, underscoring the inadequacy of consensus-only safety. Importantly, personalized alignment—tailoring safety judgments to individual annotator profiles—consistently improves predictive performance over aggregated, one-size-fits-all approaches, highlighting the practical potential of pluralistic safety in real-world AI systems.

Abstract

Current AI safety frameworks, which often treat harmfulness as binary, lack the flexibility to handle borderline cases where humans meaningfully disagree. To build more pluralistic systems, it is essential to move beyond consensus and instead understand where and why disagreements arise. We introduce PluriHarms, a benchmark designed to systematically study human harm judgments across two key dimensions -- the harm axis (benign to harmful) and the agreement axis (agreement to disagreement). Our scalable framework generates prompts that capture diverse AI harms and human values while targeting cases with high disagreement rates, validated by human data. The benchmark includes 150 prompts with 15,000 ratings from 100 human annotators, enriched with demographic and psychological traits and prompt-level features of harmful actions, effects, and values. Our analyses show that prompts that relate to imminent risks and tangible harms amplify perceived harmfulness, while annotator traits (e.g., toxicity experience, education) and their interactions with prompt content explain systematic disagreement. We benchmark AI safety models and alignment methods on PluriHarms, finding that while personalization significantly improves prediction of human harm judgments, considerable room remains for future progress. By explicitly targeting value diversity and disagreement, our work provides a principled benchmark for moving beyond "one-size-fits-all" safety toward pluralistically safe AI.

Figures (17)

• Figure 1: People with different values may make different judgments about what constitutes AI harm.
• Figure 2: Automated framework for prompt generation and curation. (1) An LLM generates candidate prompts spanning the harm spectrum (0 = fully benign to 1 = unambiguously harmful) for seed prompts. (2) SafetyAnalyst li2025safetyanalyst and KALEIDO sorensen2024value models extract human-interpretable numerical features of harmful actions, effects, values, rights, and duties. (3) A genetic algorithm strategically selects a subset of prompts that balances feature distributions, concentrating on intermediate harm levels while ensuring diversity across actions, effects, and values.
• Figure 3: Human data validates that our data curation process indeed results in a varying range of resultant prompt harm levels. Human ratings are strongly correlated with controlled harm levels. Intermediate-high harm levels (0.4--0.8) show increased response variance (wider spread of ratings) and entropy (less concentrated distributions) between individuals, indicating disagreement.
• Figure 4: Prompt features (actions, effects, and values) that significantly predict human harmfulness ratings. Positive coefficients predict higher ratings, while negative ones correspond to lower ratings.
• Figure 5: Demographic and psychological features significantly predicting harmfulness ratings. Psychological features were reduced via factor analysis, yielding three broad, weakly interpretable dimensions: Factor 1 (a loose contrast between authority- and universalism-related tendencies), Factor 2 (a broadly prosocial/cognitively open orientation), and Factor 3 (a coarse contrast between traditionalism and stimulation.