Figurative-cum-Commonsense Knowledge Infusion for Multimodal Mental Health Meme Classification

TL;DR

The paper tackles the challenge of interpreting mental health signals expressed through memes, where figurative language and common-sense knowledge are essential for understanding. It introduces AxiOM, a 3,582-meme dataset derived from the GAD anxiety questionnaire, and the M3H framework that fuses figurative reasoning with retrieval-augmented knowledge and a BART-based classifier. Through comprehensive benchmarks on both AxiOM and the RESTORE depressive-meme dataset, M3H achieves consistent improvements over a wide set of baselines, with macro-F1 gains of around 4–5 percentage points and robust ablation evidence for the knowledge-fusion components. Human evaluation supports the quality of the generated figurative reasoning, underscoring the practical impact of integrating commonsense and figurative understanding into multimodal mental-health meme analysis.

Abstract

The expression of mental health symptoms through non-traditional means, such as memes, has gained remarkable attention over the past few years, with users often highlighting their mental health struggles through figurative intricacies within memes. While humans rely on commonsense knowledge to interpret these complex expressions, current Multimodal Language Models (MLMs) struggle to capture these figurative aspects inherent in memes. To address this gap, we introduce a novel dataset, AxiOM, derived from the GAD anxiety questionnaire, which categorizes memes into six fine-grained anxiety symptoms. Next, we propose a commonsense and domain-enriched framework, M3H, to enhance MLMs' ability to interpret figurative language and commonsense knowledge. The overarching goal remains to first understand and then classify the mental health symptoms expressed in memes. We benchmark M3H against 6 competitive baselines (with 20 variations), demonstrating improvements in both quantitative and qualitative metrics, including a detailed human evaluation. We observe a clear improvement of 4.20% and 4.66% on weighted-F1 metric. To assess the generalizability, we perform extensive experiments on a public dataset, RESTORE, for depressive symptom identification, presenting an extensive ablation study that highlights the contribution of each module in both datasets. Our findings reveal limitations in existing models and the advantage of employing commonsense to enhance figurative understanding.

Figures (7)

• Figure 1: Our problem statement involves the classification of mental health memes focusing on depressive and anxious social media content. We employ figurative reasoning in our proposed method, M3H, for symptom classification.
• Figure 2: Label-wise sample of memes in the proposed dataset, AxiOM (A1 -- A6), illustrating different GAD anxious symptoms and RESTORE (D1 -- D7), illustrating different PHQ-9 depressive symptoms.
• Figure 3: Proposed Framework: M3H. The meme image and OCR text are standalone inputs to our framework. Primarily, we employ LLM for figurative reasoning on prominent commonsense attributes. We deploy the reasoning into a RAG-database to be considered for knowledge fusion during classification. Later, we employ an encoder-decoder framework for final classification.
• Figure 4: Confusion matrix computed on our proposed dataset, AxiOM using our model, M3H.
• Figure 5: Error Analysis. We attempt to connect the dots between misclassification and figurative reasoning. Evidently, the reasoning module is able to capture underlying nuances; however, we observe label confusion due to which misclassification happens.