PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements

TL;DR

PsyDI presents a novel AI-driven framework for personalized psychological measurements by combining a progressively in-depth multi-turn pipeline with a score model that ranks the indicativeness of statements for MBTI. The problem is formalized as an MBTI-oriented Markov decision process, and practical constraints are addressed with a two-phase action decomposition and modular, statement-centered dialogues. The score model is trained with a ranking-based paradigm using pairwise data and MarginRankingLoss, enabling robust estimation despite unobservable psychological indicators. Empirical results on multilingual datasets and large online deployment demonstrate superior performance over baselines and effective generalization to additional psychometrics, underscoring PsyDI’s potential as a general framework for interactive psychological assessments.

Abstract

In the field of psychology, traditional assessment methods, such as standardized scales, are frequently critiqued for their static nature, lack of personalization, and reduced participant engagement, while comprehensive counseling evaluations are often inaccessible. The complexity of quantifying psychological traits further limits these methods. Despite advances with large language models (LLMs), many still depend on single-round Question-and-Answer interactions. To bridge this gap, we introduce PsyDI, a personalized and progressively in-depth chatbot designed for psychological measurements, exemplified by its application in the Myers-Briggs Type Indicator (MBTI) framework. PsyDI leverages user-related multi-modal information and engages in customized, multi-turn interactions to provide personalized, easily accessible measurements, while ensuring precise MBTI type determination. To address the challenge of unquantifiable psychological traits, we introduce a novel training paradigm that involves learning the ranking of proxy variables associated with these traits, culminating in a robust score model for MBTI measurements. The score model enables PsyDI to conduct comprehensive and precise measurements through multi-turn interactions within a unified estimation context. Through various experiments, we validate the efficacy of both the score model and the PsyDI pipeline, demonstrating its potential to serve as a general framework for psychological measurements. Furthermore, the online deployment of PsyDI has garnered substantial user engagement, with over 3,000 visits, resulting in the collection of numerous multi-turn dialogues annotated with MBTI types, which facilitates further research. The source code for the training and web service components is publicly available as a part of OpenDILab at: https://github.com/opendilab/PsyDI

Figures (22)

• Figure 1: Comparison of traditional psychological assessment and the PsyDI framework. (left) Traditional assessment, which relies on general questions, can only discern external behavioral traits of the user. This often leads to erroneous MBTI measurements due to controversial external behaviors, such as the INFP person exhibiting some ENFP behaviors. (right) In contrast, the PsyDI framework poses questions based on familiar life scenarios, such as topics like friendship or workplace, and gradually delves into the user's internal cognitive functions through their external behaviors. This comprehensive approach allows for a more accurate identification of the underlying causes of controversial behaviors and provides a precise MBTI measurement.
• Figure 2: Pipeline Overview. PsyDI operates in a loop comprising three phases. The process begins with the user providing statements to initialize the MBTI profile. Based on the current profile, PsyDI selects a specific statement and engages in a multi-turn dialogue with the user. The interaction outcomes are used to update the profile. This iterative loop persists until PsyDI achieves high confidence in the user's MBTI.
• Figure 3: Score model training pipeline. It begins by using ChatGPT to predict the probability of each statement's MBTI type. We then construct pairs of statements based on predictions and their true labels $m$. For each pair-wise statements $(p_i, p_j, m)$, where the statement $p_i$ is predicted to align more closely with the $m$ than statement $p_j$, the loss function is defined as the difference in the predicted probabilities under the $m$ between $p_i$ and $p_j$. This ensures that the statement more accurately matching $m$ receives a higher score.
• Figure 4: Impact of iterative introversion augmentation on Score Model patterns. Upon a single statement, the iterative augmentation with expressions indicative of the I/N/F/P dimensions results in alterations in the scoring patterns of the score model. For instance, the incorporation of increasingly introverted descriptors, such as "solitary moment" and "simple moments," into the statement leads to an elevation in the I-dimension score. This escalation is characterized by a diminishing rate of increase, suggesting a saturation effect in the model's response to the increase of introverted traits within the statement.
• Figure 5: The score of sentences with extroverted (E) semantics but introverted (I) words