Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators

Sungjib Lim, Woojung Song, Eun-Ju Lee, Yohan Jo

TL;DR

This work tackles construct validity in psychometric item validation for large language models by introducing mediator-guided virtual respondents. It proposes a five-stage framework: select target traits from Big5/Schwartz/VIA, generate an expanded item pool, create mediators using several strategies, simulate responses with mediator-driven prompts, and rank items by convergent validity against official surveys, validated against human data. Across experiments, mediator-based item generation—especially trait-only mediator strategies—achieves top CV and high ranking performance, outperforming baselines and illustrating that LLMs can generate plausible mediators and simulate trait responses. The study provides a cost-effective benchmark, demonstrates robustness across models and scales, and highlights the importance of mediators for robust item validity, while releasing data and code to spur future work.

Abstract

As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs. Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that robustly measure intended traits. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent behavior for item validation. Our problem formulation, metrics, methodology, and dataset open a new direction for cost-effective survey development and a deeper understanding of how LLMs simulate human survey responses. We publicly release our dataset and code to support future work.

Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators

TL;DR

This work tackles construct validity in psychometric item validation for large language models by introducing mediator-guided virtual respondents. It proposes a five-stage framework: select target traits from Big5/Schwartz/VIA, generate an expanded item pool, create mediators using several strategies, simulate responses with mediator-driven prompts, and rank items by convergent validity against official surveys, validated against human data. Across experiments, mediator-based item generation—especially trait-only mediator strategies—achieves top CV and high ranking performance, outperforming baselines and illustrating that LLMs can generate plausible mediators and simulate trait responses. The study provides a cost-effective benchmark, demonstrates robustness across models and scales, and highlights the importance of mediators for robust item validity, while releasing data and code to spur future work.

Abstract

As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs. Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that robustly measure intended traits. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent behavior for item validation. Our problem formulation, metrics, methodology, and dataset open a new direction for cost-effective survey development and a deeper understanding of how LLMs simulate human survey responses. We publicly release our dataset and code to support future work.

Paper Structure

This paper contains 73 sections, 7 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Psychological Assessment for LLMs and Humans.
  4. Survey Expansion and Reduction.
  5. Validity and Reliability.
  6. Simulating Human Behavior Using LLMs.
  7. Mediators.
  8. Framework
  9. Traits Selection (Figure \ref{['fig_overview']}-A)
  10. Item Generation (Figure \ref{['fig_overview']}-B)
  11. Mediator Generation (Figure \ref{['fig_overview']}-C)
  12. Mediator-Guided Simulation (Figure \ref{['fig_overview']}-D)
  13. Target trait.
  14. Mediator-integrated persona profile.
  15. Survey item, answer choices, and instructions.
  16. ...and 58 more sections

Figures (9)

  • Figure 1: Role of mediators in survey item evaluation.
  • Figure 2: Overview of our framework.
  • Figure 3: Example prompt used in mediator-guided simulation for Big5. Blue text is the mediator. Texts highlighted in green are not included in actual prompts.
  • Figure 4: Mediator categories.
  • Figure 5: Performance of selected items across different components of the simulation prompt. Percentile scores were normalized to the [0, 1] range.
  • ...and 4 more figures