PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection

TL;DR

PsyCoT reframes personality detection as a multi-turn reasoning task by embedding psychological questionnaire items as explicit chain-of-thought steps for an LLM. By rating items sequentially and aggregating results, the approach improves zero-shot performance on Big Five and MBTI tasks, achieving notable gains over standard prompting and approaching fine-tuned baselines. The study demonstrates robustness to prompt variations and highlights the importance of questionnaire depth, while also discussing limitations and ethical implications. Overall, PsyCoT offers a practical, data-efficient method to leverage LLM reasoning in personality inference from text.

Abstract

Recent advances in large language models (LLMs), such as ChatGPT, have showcased remarkable zero-shot performance across various NLP tasks. However, the potential of LLMs in personality detection, which involves identifying an individual's personality from their written texts, remains largely unexplored. Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, we argue that these items can be regarded as a collection of well-structured chain-of-thought (CoT) processes. By incorporating these processes, LLMs can enhance their capabilities to make more reasonable inferences on personality from textual input. In light of this, we propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. In particular, we employ a LLM as an AI assistant with a specialization in text analysis. We prompt the assistant to rate individual items at each turn and leverage the historical rating results to derive a conclusive personality preference. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection, achieving an average F1 score improvement of 4.23/10.63 points on two benchmark datasets compared to the standard prompting method. Our code is available at https://github.com/TaoYang225/PsyCoT.

Figures (7)

• Figure 1: An illustration of our PsyCoT, where items (1-8) from the personality questionnaire are employed as the CoT to answer the final personality inquiry. We prompt the LLM rating items based on the author's text, which simulates the process of human to complete personality tests through a multi-turn dialogue.
• Figure 2: Comparison of Standard Prompting (Top) and our PsyCoT Prompting (Bottom). In PsyCoT, the dotted box indicates a reasoning step derived from a psychological questionnaire. Unlike Standard Prompting, which directly prompts LLM to output the personality preference, PsyCoT incorporates multiple reasoning steps through interactions with the LLM, guiding the LLM to infer personality in a more reasonable manner.
• Figure 3: Distributions of the trait scores across five personalities in the Essays dataset. The dashed line represents the neutral value defined by the questionnaire (i.e., 3=Neutral). The value of "S" displayed in figures indicate the Spearman's Coefficient between trait scores and the personality dimensions. A value closer to 1 suggests a stronger positive correlation. Results demonstrate a strong positive correlation between trait scores and personality type, particularly in Agreeableness, Extraversion, and Openness.
• Figure 4: The robustness testing results for three prompt-based methods on the five traits. PsyCoT demonstrates the highest unchanged rate, indicating its robustness in handling option order perturbations.
• Figure 5: Results of the study on post orders. The three prompt-based methods are influenced by post orders for most of personality traits.