FITMM: Adaptive Frequency-Aware Multimodal Recommendation via Information-Theoretic Representation Learning
Wei Yang, Rui Zhong, Yiqun Chen, Shixuan Li, Heng Ping, Chi Lu, Peng Jiang
TL;DR
Multimodal recommender systems often fuse heterogeneous signals in the spatial domain, causing misalignment and redundancy. FITMM introduces a frequency-aware information-theoretic framework that decomposes modalities into frequency bands, performs bandwise cross-modal fusion, and applies an information bottleneck and spectral-consistency regularization; theory shows that under an orthogonal transform the Gaussian IB decouples across bands, enabling a separate-then-fuse design with Wiener-like per-band gating ($\mathcal{L}_{IB}=\sum_{k=1}^K \mathcal{L}^{(k)}_{IB}$). The approach uses graph-enhanced, modality-specific representations, per-band interactions, and a global budget to allocate capacity, achieving strong empirical gains on three real-world datasets and particularly improving cold-start recommendations. Ablation and spectral analyses corroborate the importance of frequency decomposition and IB-based regularization for robust, interpretable multimodal personalization.
Abstract
Multimodal recommendation aims to enhance user preference modeling by leveraging rich item content such as images and text. Yet dominant systems fuse modalities in the spatial domain, obscuring the frequency structure of signals and amplifying misalignment and redundancy. We adopt a spectral information-theoretic view and show that, under an orthogonal transform that approximately block-diagonalizes bandwise covariances, the Gaussian Information Bottleneck objective decouples across frequency bands, providing a principled basis for separate-then-fuse paradigm. Building on this foundation, we propose FITMM, a Frequency-aware Information-Theoretic framework for multimodal recommendation. FITMM constructs graph-enhanced item representations, performs modality-wise spectral decomposition to obtain orthogonal bands, and forms lightweight within-band multimodal components. A residual, task-adaptive gate aggregates bands into the final representation. To control redundancy and improve generalization, we regularize training with a frequency-domain IB term that allocates capacity across bands (Wiener-like shrinkage with shut-off of weak bands). We further introduce a cross-modal spectral consistency loss that aligns modalities within each band. The model is jointly optimized with the standard recommendation loss. Extensive experiments on three real-world datasets demonstrate that FITMM consistently and significantly outperforms advanced baselines.