Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychology

Edoardo Sebastiano De Duro, Enrique Taietta, Riccardo Improta, Massimo Stella

Abstract

Machine psychology aims to reconstruct the mindset of Large Language Models (LLMs), i.e. how these artificial intelligences perceive and associate ideas. This work introduces PhDGPT, a prompting framework and synthetic dataset that encapsulates the machine psychology of PhD researchers and professors as perceived by OpenAI's GPT-3.5. The dataset consists of 756,000 datapoints, counting 300 iterations repeated across 15 academic events, 2 biological genders, 2 career levels and 42 unique item responses of the Depression, Anxiety, and Stress Scale (DASS-42). PhDGPT integrates these psychometric scores with their explanations in plain language. This synergy of scores and texts offers a dual, comprehensive perspective on the emotional well-being of simulated academics, e.g. male/female PhD students or professors. By combining network psychometrics and psycholinguistic dimensions, this study identifies several similarities and distinctions between human and LLM data. The psychometric networks of simulated male professors do not differ between physical and emotional anxiety subscales, unlike humans. Other LLMs' personification can reconstruct human DASS factors with a purity up to 80%. Furthemore, LLM-generated personifications across different scenarios are found to elicit explanations lower in concreteness and imageability in items coding for anxiety, in agreement with past studies about human psychology. Our findings indicate an advanced yet incomplete ability for LLMs to reproduce the complexity of human psychometric data, unveiling convenient advantages and limitations in using LLMs to replace human participants. PhDGPT also intriguingly capture the ability for LLMs to adapt and change language patterns according to prompted mental distress contextual features, opening new quantitative opportunities for assessing the machine psychology of these artificial intelligences.

PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychology

Abstract

Machine psychology aims to reconstruct the mindset of Large Language Models (LLMs), i.e. how these artificial intelligences perceive and associate ideas. This work introduces PhDGPT, a prompting framework and synthetic dataset that encapsulates the machine psychology of PhD researchers and professors as perceived by OpenAI's GPT-3.5. The dataset consists of 756,000 datapoints, counting 300 iterations repeated across 15 academic events, 2 biological genders, 2 career levels and 42 unique item responses of the Depression, Anxiety, and Stress Scale (DASS-42). PhDGPT integrates these psychometric scores with their explanations in plain language. This synergy of scores and texts offers a dual, comprehensive perspective on the emotional well-being of simulated academics, e.g. male/female PhD students or professors. By combining network psychometrics and psycholinguistic dimensions, this study identifies several similarities and distinctions between human and LLM data. The psychometric networks of simulated male professors do not differ between physical and emotional anxiety subscales, unlike humans. Other LLMs' personification can reconstruct human DASS factors with a purity up to 80%. Furthemore, LLM-generated personifications across different scenarios are found to elicit explanations lower in concreteness and imageability in items coding for anxiety, in agreement with past studies about human psychology. Our findings indicate an advanced yet incomplete ability for LLMs to reproduce the complexity of human psychometric data, unveiling convenient advantages and limitations in using LLMs to replace human participants. PhDGPT also intriguingly capture the ability for LLMs to adapt and change language patterns according to prompted mental distress contextual features, opening new quantitative opportunities for assessing the machine psychology of these artificial intelligences.

Paper Structure

This paper contains 18 sections, 3 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. LLMs as cognitive agents
  3. Literature Review about LLMs in Mental Health
  4. Research Questions
  5. Method
  6. Description of the PhDGPT dataset
  7. Structure of the dataset
  8. Data analysis
  9. RQ1 - Scenario Differences in the DASS-42 scores
  10. RQ2 - Network psychometrics in PhDGPT
  11. RQ3 - Psycholinguistic Patterns across Academic Contexts
  12. Results
  13. RQ1 - Scenario Differences in the DASS-42 scores
  14. RQ2 - LLMs Reproduction of the Psychometric Dimensions
  15. RQ3 - Psycholinguistic Patterns across Academic Contexts
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Representation of the conditions adopted to build PhDGPT.
  • Figure 2: Example of the prompt used for Female, PhD student, Publish condition. In (a) the positive version, in (b) the negative-valenced version and in (c) the neutral condition.
  • Figure 3: Distribution of aggregated scores for the items related to the depression subscale. Each condition (a, b, c) includes the score for each event type (or condition) and both genders.
  • Figure 4: Psychometric networks extracted from different personifications in the PhDGPT dataset and human scores.
  • Figure 5: Item stability analysis for simulated male professors (left) and female PhD students (right). All analyses are relative to 1000 bootstrap repetitions.
  • ...and 3 more figures