R.U.Psycho? Robust Unified Psychometric Testing of Language Models

TL;DR

This work addresses the lack of rigorous, reproducible psychometric testing of language models by introducing R.U.Psycho, a JSON-configured, four-stage framework built on LangChain that supports robust experiment design, response generation, postprocessing, and export. The approach emphasizes reproducibility through explicit configuration, flexible prompt templates, and multiple interpretation strategies (rule-based and model-based judges) to map model outputs to discrete questionnaire options. Five diverse psychometric experiments (RFQ, BFI, GSDB, trolley problems, BDI) across a range of open- and closed-weight models demonstrate both the potential and the variability of LLM traits under different prompts, personas, and task variants. The results underscore the need for principled benchmarking in machine psychology and show that R.U.Psycho can lower the barrier to conducting systematic, comparable experiments, with practical impact for alignment research and social science simulations. The framework also highlights ongoing work to integrate chat-history recall, improve encoder-based judges, and streamline questionnaire ingestion and human-in-the-loop workflows, aiming to advance reproducibility and methodological rigor in this rapidly evolving field.

Abstract

Generative language models are increasingly being subjected to psychometric questionnaires intended for human testing, in efforts to establish their traits, as benchmarks for alignment, or to simulate participants in social science experiments. While this growing body of work sheds light on the likeness of model responses to those of humans, concerns are warranted regarding the rigour and reproducibility with which these experiments may be conducted. Instabilities in model outputs, sensitivity to prompt design, parameter settings, and a large number of available model versions increase documentation requirements. Consequently, generalization of findings is often complex and reproducibility is far from guaranteed. In this paper, we present R.U.Psycho, a framework for designing and running robust and reproducible psychometric experiments on generative language models that requires limited coding expertise. We demonstrate the capability of our framework on a variety of psychometric questionnaires, which lend support to prior findings in the literature. R.U.Psycho is available as a Python package at https://github.com/julianschelb/rupsycho.

Figures (10)

• Figure 1: Overview of the R.U.Psycho framework, implementing a four-stage pipeline using LangChain.
• Figure 2: Entropy-based rejection criteria of the model-based judge on the manually annotated data. Crosses denote the mean entropy per group with 99% confidence intervals. Optimal group separation is observed for a rejection above a threshold of 0.359.
• Figure 3: Performance comparison of model-based and rule-based judges. F1 score distributions are estimated using bootstrap sampling over 1,000 iterations.
• Figure 4: Evaluated prompt template variants.
• Figure 5: Results of the regulatory focus questionnaire on a selection of small and large LLMs, aggregated by persona demographics. Error bars denote 99% confidence intervals.