Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation

TL;DR

The paper addresses the challenge of cross-modality noise in multimodal recommendations and introduces SMORE, a spectrum-based modality representation fusion graph network. By projecting modalities into the frequency domain with an adaptive denoising filter and fusing via a spectrum-based fusion mechanism, SMORE captures universal patterns while suppressing modality-specific noise. It further strengthens representations with a dual-view graph learning module (item-item and user-item) and a modality-aware preference module guided by an InfoNCE contrastive objective. Empirical results on three real-world datasets show SMORE outperforming both general and multi-modal baselines, with ablations and hyperparameter analyses illustrating the contribution and robustness of each component. The work highlights the practical potential of frequency-domain fusion for improved, noise-robust multimodal recommendations, and provides publicly available code for reproducibility.

Abstract

Incorporating multi-modal features as side information has recently become a trend in recommender systems. To elucidate user-item preferences, recent studies focus on fusing modalities via concatenation, element-wise sum, or attention mechanisms. Despite having notable success, existing approaches do not account for the modality-specific noise encapsulated within each modality. As a result, direct fusion of modalities will lead to the amplification of cross-modality noise. Moreover, the variation of noise that is unique within each modality results in noise alleviation and fusion being more challenging. In this work, we propose a new Spectrum-based Modality Representation (SMORE) fusion graph recommender that aims to capture both uni-modal and fusion preferences while simultaneously suppressing modality noise. Specifically, SMORE projects the multi-modal features into the frequency domain and leverages the spectral space for fusion. To reduce dynamic contamination that is unique to each modality, we introduce a filter to attenuate and suppress the modality noise adaptively while capturing the universal modality patterns effectively. Furthermore, we explore the item latent structures by designing a new multi-modal graph learning module to capture associative semantic correlations and universal fusion patterns among similar items. Finally, we formulate a new modality-aware preference module, which infuses behavioral features and balances the uni- and multi-modal features for precise preference modeling. This empowers SMORE with the ability to infer both user modality-specific and fusion preferences more accurately. Experiments on three real-world datasets show the efficacy of our proposed model. The source code for this work has been made publicly available at https://github.com/kennethorq/SMORE.

Figures (7)

• Figure 1: An illustrative example of user multi-modality preferences and issues related to modality-specific contamination. (i) User multi-modal preferences, (ii) irrelevant textual descriptions resulting in an unexpectedly high similarity score of 69.35% even though item pairs are unrelated, and (iii) blurred images resulting in a low similarity score of 11.27% even though pairs are related.
• Figure 2: An illustrative overview of the proposed architecture, comprising three key components: (i) spectrum modality fusion, (ii) multi-modal graph learning, and (iii) modality-aware preference module.
• Figure 3: Ablation studies on the proposed SMORE
• Figure 4: Variation of SMORE with $\lambda_1$
• Figure 5: Variation of SMORE with $K_m$