Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment

Nuo Chen, Hanpei Fang, Piaohong Wang, Jiqun Liu, Tetsuya Sakai, Xiao-Ming Wu

TL;DR

The study addresses how prompting LLMs to simulate Big Five personality traits influences relevance judgments and confidence calibration in information retrieval. It introduces PERAS, a framework that generates eleven personality-infused assessors (ten high/low variants plus a default), collects relevance labels and self-reported confidences, and uses ML to aggregate signals across personalities. Key findings show that Low Agreeableness yields the strongest alignment with human judgments, while Low Conscientiousness and High Neuroticism balance calibration, and that combining scores and confidences from multiple personalities via Random Forest outperforms the best single persona, even with limited labeled data. The work advances a psychologically grounded, confidence-aware approach to automated IR evaluation, suggesting that personality-driven uncertainty signals can improve reliability and interpretability of LLM-based judgment systems, with implications for multi-agent personality interactions and robust ground-truth estimation.

Abstract

Recent studies have shown that prompting can enable large language models (LLMs) to simulate specific personality traits and produce behaviors that align with those traits. However, there is limited understanding of how these simulated personalities influence critical web search decisions, specifically relevance assessment. Moreover, few studies have examined how simulated personalities impact confidence calibration, specifically the tendencies toward overconfidence or underconfidence. This gap exists even though psychological literature suggests these biases are trait-specific, often linking high extraversion to overconfidence and high neuroticism to underconfidence. To address this gap, we conducted a comprehensive study evaluating multiple LLMs, including commercial models and open-source models, prompted to simulate Big Five personality traits. We tested these models across three test collections (TREC DL 2019, TREC DL 2020, and LLMJudge), collecting two key outputs for each query-document pair: a relevance judgment and a self-reported confidence score. The findings show that personalities such as low agreeableness consistently align more closely with human labels than the unprompted condition. Additionally, low conscientiousness performs well in balancing the suppression of both overconfidence and underconfidence. We also observe that relevance scores and confidence distributions vary systematically across different personalities. Based on the above findings, we incorporate personality-conditioned scores and confidence as features in a random forest classifier. This approach achieves performance that surpasses the best single-personality condition on a new dataset (TREC DL 2021), even with limited training data. These findings highlight that personality-derived confidence offers a complementary predictive signal, paving the way for more reliable and human-aligned LLM evaluators.

Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment

TL;DR

The study addresses how prompting LLMs to simulate Big Five personality traits influences relevance judgments and confidence calibration in information retrieval. It introduces PERAS, a framework that generates eleven personality-infused assessors (ten high/low variants plus a default), collects relevance labels and self-reported confidences, and uses ML to aggregate signals across personalities. Key findings show that Low Agreeableness yields the strongest alignment with human judgments, while Low Conscientiousness and High Neuroticism balance calibration, and that combining scores and confidences from multiple personalities via Random Forest outperforms the best single persona, even with limited labeled data. The work advances a psychologically grounded, confidence-aware approach to automated IR evaluation, suggesting that personality-driven uncertainty signals can improve reliability and interpretability of LLM-based judgment systems, with implications for multi-agent personality interactions and robust ground-truth estimation.

Abstract

Recent studies have shown that prompting can enable large language models (LLMs) to simulate specific personality traits and produce behaviors that align with those traits. However, there is limited understanding of how these simulated personalities influence critical web search decisions, specifically relevance assessment. Moreover, few studies have examined how simulated personalities impact confidence calibration, specifically the tendencies toward overconfidence or underconfidence. This gap exists even though psychological literature suggests these biases are trait-specific, often linking high extraversion to overconfidence and high neuroticism to underconfidence. To address this gap, we conducted a comprehensive study evaluating multiple LLMs, including commercial models and open-source models, prompted to simulate Big Five personality traits. We tested these models across three test collections (TREC DL 2019, TREC DL 2020, and LLMJudge), collecting two key outputs for each query-document pair: a relevance judgment and a self-reported confidence score. The findings show that personalities such as low agreeableness consistently align more closely with human labels than the unprompted condition. Additionally, low conscientiousness performs well in balancing the suppression of both overconfidence and underconfidence. We also observe that relevance scores and confidence distributions vary systematically across different personalities. Based on the above findings, we incorporate personality-conditioned scores and confidence as features in a random forest classifier. This approach achieves performance that surpasses the best single-personality condition on a new dataset (TREC DL 2021), even with limited training data. These findings highlight that personality-derived confidence offers a complementary predictive signal, paving the way for more reliable and human-aligned LLM evaluators.
Paper Structure (27 sections, 9 equations, 5 figures, 9 tables)

This paper contains 27 sections, 9 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs as Relevance Assessors
  4. Personality and Confidence
  5. LLMs Infusing Personality
  6. Confidence Calibration of LLMs
  7. Research Questions
  8. Methodology
  9. Personality Infusion
  10. Relevance Assessment Procedure
  11. Confidence Rating Mechanism
  12. Machine Learning-Based Relevance Labelling
  13. Experiments
  14. Datasets and Backbone LLMs
  15. Experimental Setup
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Schematic representation of the experimental pipeline for personality-simulated relevance assessment. The process begins with the transformation of Big Five personality traits into descriptive instructions. Experiment 1 evaluates the performance and self-calibration of a single personality-infused agent by capturing its task predictions and associated confidence levels. Experiment 2 extends this to a multi-agent setting, where outputs from assessors with varying personalities are utilized as features for a downstream Machine Learning (ML) classifier to derive the final consensus prediction.
  • Figure 2: Cohen’s Kappa heatmaps across personality conditions simulated by GPT-4o and Llama-3-70b on the LLMJudge dataset.
  • Figure 3: Average confidence for predictions of score = 0 across ground truth labels, comparing high neuroticism (HN) and low conscientiousness (LC) personality conditions simulated by GPT-4o on LLMJudge.
  • Figure 4: Comparison of evaluation metrics ($\kappa$, QWK, F1) across different personality dimensions and model configurations. Each radar chart shows performance across 11 personality configurations (Df: Default, HA/LA: High/Low Agreeableness, HC/LC: High/Low Conscientiousness, HE/LE: High/Low Extraversion, HN/LN: High/Low Neuroticism, HO/LO: High/Low Openness) for three datasets (LLMJudge, TRDL19, TRDL20).
  • Figure 5: Comparison of RO, RU, and HMR metrics across different personality dimensions and model configurations. Each radar chart shows performance across 11 personality configurations (Df: Default, HA/LA: High/Low Agreeableness, HC/LC: High/Low Conscientiousness, HE/LE: High/Low Extraversion, HN/LN: High/Low Neuroticism, HO/LO: High/Low Openness) for three datasets (LLMJudge, TRDL19, TRDL20). RO measures rank order correlation, RU measures rank utility, and HMR measures human-machine relevance agreement.