Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

TL;DR

This work tackles the challenge of evaluating AI systems in domains demanding ethical judgment by proposing a psychometric framework that combines domain-specific situational judgment tests (SJTs) with richly grounded police personas. The authors integrate industrial-organizational psychology and HEXACO personality theory to create trait-aligned, demographically realistic personas generated via structured pipelines, memoir grounding, and Pydantic schemas, enabling nuanced probing of AI behavior across 8 archetypes and 11 attributes. They demonstrate scalability through a law-enforcement case study producing 8,500 personas, 4,000 SJTs, and 300,000 responses, and show that HEXACO traits strongly predict SJT behavior (R^2 ≈ 0.8–0.9) with meaningful item- and trait-level patterns (e.g., r ≈ 0.504 for eXtraversion). The dataset and tooling are released publicly, and the framework is designed for extensibility to other domains, offering a scalable, domain-relevant approach to AI psychometrics that can inform ethics, safety, and governance in AI deployments.

Abstract

AI psychometrics evaluates AI systems in roles that traditionally require emotional judgment and ethical consideration. Prior work often reuses human trait inventories (Big Five, \hexaco) or ad hoc personas, limiting behavioral realism and domain relevance. We propose a framework that (1) uses situational judgment tests (SJTs) from realistic scenarios to probe domain-specific competencies; (2) integrates industrial-organizational and personality psychology to design sophisticated personas which include behavioral and psychological descriptors, life history, and social and emotional functions; and (3) employs structured generation with population demographic priors and memoir inspired narratives, encoded with Pydantic schemas. In a law enforcement assistant case study, we construct a rich dataset of personas drawn across 8 persona archetypes and SJTs across 11 attributes, and analyze behaviors across subpopulation and scenario slices. The dataset spans 8,500 personas, 4,000 SJTs, and 300,000 responses. We will release the dataset and all code to the public.

Figures (14)

• Figure 1: Overview of the synthetic persona generation pipeline. We use GPT-4.1 to generate demographically diverse police officer personas by first creating detailed memoir accounts and using them to create a comprehensive background for the persona.
• Figure 2: Overview of the scenario generation pipeline. Domain experts first develop base scenarios with trait-mapped response options (step 1). An LLM then generates scenario variants through controlled attribute manipulation (step 2), followed by iterative quality refinement to identify and correct trait bleed instances. The final dataset contains response options that are uniquely aligned to the six HEXACO traits (step 3).
• Figure 3:
• Figure 4:
• Figure 5: