Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysis

Xueqi Ma, Yanbei Jiang, Sarah Erfani, James Bailey, Weifeng Liu, Krista A. Ehinger, Jey Han Lau

TL;DR

The paper tackles the challenge of performing subjective psychological analysis from visual drawings with Multimodal Large Language Models by introducing PICK, a zero-shot framework that decomposes House-Tree-Person drawings into single-object, multi-object, and whole-level sub-drawings. It couples a hierarchical analysis pipeline with an emotion-aware reward model and a knowledge base to generate object-specific features and align reasoning with expert interpretations, then fuses multi-level signals for final mental-state predictions. Across two HTP datasets and two emotion datasets, PICK demonstrates superior diagnostic capability for mental health indicators and robust emotion understanding, while also offering improved interpretability through its multi-level visual reasoning. The approach bridges MLLMs with clinical psychology by embedding domain knowledge and test-time learning, enabling structured, explainable psychoanalytic reasoning from drawings with potential for early screening and educational use under appropriate safeguards.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance across various objective multimodal perception tasks, yet their application to subjective, emotionally nuanced domains, such as psychological analysis, remains largely unexplored. In this paper, we introduce PICK, a multi-step framework designed for Psychoanalytical Image Comprehension through hierarchical analysis and Knowledge injection with MLLMs, specifically focusing on the House-Tree-Person (HTP) Test, a widely used psychological assessment in clinical practice. First, we decompose drawings containing multiple instances into semantically meaningful sub-drawings, constructing a hierarchical representation that captures spatial structure and content across three levels: single-object level, multi-object level, and whole level. Next, we analyze these sub-drawings at each level with a targeted focus, extracting psychological or emotional insights from their visual cues. We also introduce an HTP knowledge base and design a feature extraction module, trained with reinforcement learning, to generate a psychological profile for single-object level analysis. This profile captures both holistic stylistic features and dynamic object-specific features (such as those of the house, tree, or person), correlating them with psychological states. Finally, we integrate these multi-faceted information to produce a well-informed assessment that aligns with expert-level reasoning. Our approach bridges the gap between MLLMs and specialized expert domains, offering a structured and interpretable framework for understanding human mental states through visual expression. Experimental results demonstrate that the proposed PICK significantly enhances the capability of MLLMs in psychological analysis. It is further validated as a general framework through extensions to emotion understanding tasks.

Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysis

TL;DR

The paper tackles the challenge of performing subjective psychological analysis from visual drawings with Multimodal Large Language Models by introducing PICK, a zero-shot framework that decomposes House-Tree-Person drawings into single-object, multi-object, and whole-level sub-drawings. It couples a hierarchical analysis pipeline with an emotion-aware reward model and a knowledge base to generate object-specific features and align reasoning with expert interpretations, then fuses multi-level signals for final mental-state predictions. Across two HTP datasets and two emotion datasets, PICK demonstrates superior diagnostic capability for mental health indicators and robust emotion understanding, while also offering improved interpretability through its multi-level visual reasoning. The approach bridges MLLMs with clinical psychology by embedding domain knowledge and test-time learning, enabling structured, explainable psychoanalytic reasoning from drawings with potential for early screening and educational use under appropriate safeguards.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance across various objective multimodal perception tasks, yet their application to subjective, emotionally nuanced domains, such as psychological analysis, remains largely unexplored. In this paper, we introduce PICK, a multi-step framework designed for Psychoanalytical Image Comprehension through hierarchical analysis and Knowledge injection with MLLMs, specifically focusing on the House-Tree-Person (HTP) Test, a widely used psychological assessment in clinical practice. First, we decompose drawings containing multiple instances into semantically meaningful sub-drawings, constructing a hierarchical representation that captures spatial structure and content across three levels: single-object level, multi-object level, and whole level. Next, we analyze these sub-drawings at each level with a targeted focus, extracting psychological or emotional insights from their visual cues. We also introduce an HTP knowledge base and design a feature extraction module, trained with reinforcement learning, to generate a psychological profile for single-object level analysis. This profile captures both holistic stylistic features and dynamic object-specific features (such as those of the house, tree, or person), correlating them with psychological states. Finally, we integrate these multi-faceted information to produce a well-informed assessment that aligns with expert-level reasoning. Our approach bridges the gap between MLLMs and specialized expert domains, offering a structured and interpretable framework for understanding human mental states through visual expression. Experimental results demonstrate that the proposed PICK significantly enhances the capability of MLLMs in psychological analysis. It is further validated as a general framework through extensions to emotion understanding tasks.
Paper Structure (33 sections, 3 equations, 5 figures, 13 tables)

This paper contains 33 sections, 3 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Large Language Models
  4. Drawing Psychoanalysis
  5. PICK
  6. Hierarchical Drawing Decomposition
  7. Single-Object Level Analysis
  8. Object-Specific Features Generation by a Fine-tuned MLLM
  9. Emotion-Preference Reward Model
  10. Object-Specific Features Extraction
  11. Prediction for Single-Object Level Sub-drawings
  12. Multi-Object Level and Whole-Level Analysis
  13. Prediction Based on Multi-level Analysis
  14. Experiments
  15. Experimental Settings and Evaluation Metrics
  16. ...and 18 more sections

Figures (5)

  • Figure 1: (a) An HTP drawing contains multi-level information related to mental health including single-object (S-Obj), multi-object (M-Obj) and whole levels. Individual elements, such as the house, with attributes like being empty or having an open door, may reflect feelings of loneliness or openness. At the whole level, features like shadowing may suggest isolation. (b) Existing MLLMs can identify key elements (bold text) but struggle to capture the critical information and link it to accurate psychological or mental states (red text).
  • Figure 2: The framework of PICK, including (1) Input Decomposition: The input drawing is divided into multiple sub-drawings. (2) Multi-Level Analysis: This includes tailored prompts for whole level and multi-object level analysis, a specialized feature extraction module, and the integration of a knowledge base (KB) for single-object level analysis. (3) Combined Prediction: Multi-level information is aggregated to produce a final prediction.
  • Figure 3: The visualization of PICK with multi-level analysis.
  • Figure 4: The visualization of PICK with multi-level analysis on an negative example correctly predicted as negative.
  • Figure 5: The visualization of PICK with multi-level analysis on an negative example incorrectly classified as positive.