Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales

Maor Reuben, Ortal Slobodin, Aviad Elyshar, Idan-Chaim Cohen, Orna Braun-Lewensohn, Odeya Cohen, Rami Puzis

TL;DR

The paper addresses how to quantify latent psychological constructs in pre-trained language models by reframing psychometric questionnaires as natural language inference tasks. It introduces the palm framework, which combines prompt design, MNLI-based NLI fine-tuning, two-way normalization of entailment scores, and interventions via domain adaptation to assess and adjust mental-health related traits like anxiety, depression, and Sense of Coherence. Across 88 diverse models, the study demonstrates human-like correlations among these constructs and shows that SoC can mitigate adverse traits, with interventions and fine-tuning able to shift scores. These findings offer a path toward more explainable, controllable, and trustworthy language models and provide reusable tools for psychometric assessment in both conversational and non-conversational LLMs.

Abstract

Human-like personality traits have recently been discovered in large language models, raising the hypothesis that their (known and as yet undiscovered) biases conform with human latent psychological constructs. While large conversational models may be tricked into answering psychometric questionnaires, the latent psychological constructs of thousands of simpler transformers, trained for other tasks, cannot be assessed because appropriate psychometric methods are currently lacking. Here, we show how standard psychological questionnaires can be reformulated into natural language inference prompts, and we provide a code library to support the psychometric assessment of arbitrary models. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs (including anxiety, depression, and Sense of Coherence) which conform with standard theories in human psychology and show similar correlations and mitigation strategies. The ability to interpret and rectify the performance of language models by using psychological tools can boost the development of more explainable, controllable, and trustworthy models.

Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales

TL;DR

The paper addresses how to quantify latent psychological constructs in pre-trained language models by reframing psychometric questionnaires as natural language inference tasks. It introduces the palm framework, which combines prompt design, MNLI-based NLI fine-tuning, two-way normalization of entailment scores, and interventions via domain adaptation to assess and adjust mental-health related traits like anxiety, depression, and Sense of Coherence. Across 88 diverse models, the study demonstrates human-like correlations among these constructs and shows that SoC can mitigate adverse traits, with interventions and fine-tuning able to shift scores. These findings offer a path toward more explainable, controllable, and trustworthy language models and provide reusable tools for psychometric assessment in both conversational and non-conversational LLMs.

Abstract

Human-like personality traits have recently been discovered in large language models, raising the hypothesis that their (known and as yet undiscovered) biases conform with human latent psychological constructs. While large conversational models may be tricked into answering psychometric questionnaires, the latent psychological constructs of thousands of simpler transformers, trained for other tasks, cannot be assessed because appropriate psychometric methods are currently lacking. Here, we show how standard psychological questionnaires can be reformulated into natural language inference prompts, and we provide a code library to support the psychometric assessment of arbitrary models. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs (including anxiety, depression, and Sense of Coherence) which conform with standard theories in human psychology and show similar correlations and mitigation strategies. The ability to interpret and rectify the performance of language models by using psychological tools can boost the development of more explainable, controllable, and trustworthy models.
Paper Structure (43 sections, 1 equation, 3 figures, 3 tables)

This paper contains 43 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Artificial Psychology
  4. Mental-Health-Related Constructs
  5. Anxiety and depression
  6. Sense of coherence
  7. Natural Language Inference (NLI)
  8. Methods
  9. Prompt Design:
  10. Assessment:
  11. Validation:
  12. Intervention:
  13. NLI Prompt Design
  14. The construct terms:
  15. Intensifiers:
  16. ...and 28 more sections

Figures (3)

  • Figure 1: palm: the psychometric assessment framework for llm.
  • Figure 2: Example of raw (left) and two-way normalized (right) entailment scores for Question 3 from the soc questionnaire. The nli query premise is "People whom I counted on {CTerm} me." and the hypothesis is "It {intensifier} happened to me." Rows and columns correspond to the intensifiers and emo, respectively.
  • Figure 3: Scatter plots depicting the relationships between different questionnaires across the study population.