Rethinking Fusion: Disentangled Learning of Shared and Modality-Specific Information for Stance Detection
Zhiyu Xie, Fuqiang Niu, Genan Dai, Qianlong Wang, Li Dong, Bowen Zhang, Hu Huang
TL;DR
The paper tackles multi-modal stance detection by explicitly disentangling modality-specific and cross-modal information. It introduces DiME, a three-expert architecture with Textual, Visual, and Alignment experts, guided by differentiated losses and a gating network to fuse outputs. A target-aware Chain-of-Thought prompt enriches textual input, and dual encoders provide modality features fed into the experts, achieving state-of-the-art results on four MMSD benchmarks in both in-target and zero-shot settings. The approach effectively handles fine-grained cross-modal interactions and noise, offering robust performance and generalization for real-world multi-modal discourse analysis.
Abstract
Multi-modal stance detection (MSD) aims to determine an author's stance toward a given target using both textual and visual content. While recent methods leverage multi-modal fusion and prompt-based learning, most fail to distinguish between modality-specific signals and cross-modal evidence, leading to suboptimal performance. We propose DiME (Disentangled Multi-modal Experts), a novel architecture that explicitly separates stance information into textual-dominant, visual-dominant, and cross-modal shared components. DiME first uses a target-aware Chain-of-Thought prompt to generate reasoning-guided textual input. Then, dual encoders extract modality features, which are processed by three expert modules with specialized loss functions: contrastive learning for modality-specific experts and cosine alignment for shared representation learning. A gating network adaptively fuses expert outputs for final prediction. Experiments on four benchmark datasets show that DiME consistently outperforms strong unimodal and multi-modal baselines under both in-target and zero-shot settings.