Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation

TL;DR

A preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire with the validated human-developed version shows differences emerged in dimensionality and in the distribution of item and test information across latent traits.

Abstract

Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools.

Figures (12)

• Figure 1: Frequency distribution of response levels for each item in the original BAQ questionnaire (upper-left) and the ChatGPT-generated version v1 (upper-right) and v2 (lower-left).
• Figure 2: MCMC posterior distribution for $\beta$ difficulty parameters.
• Figure 3: MCMC posterior distribution for $\gamma$ discrimination parameters.
• Figure 4: MCMC posterior distribution for $\delta$ level thresholds.
• Figure 5: Box plots of MCMC posterior distributions for $\beta$ difficulty parameters. For each test, the parameters have been sorted in increasing order.