Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

TL;DR

The paper addresses cross-modal heterogeneity in image-text learning by proposing MARNet, a plug-and-play framework with two core components: Embedding Matching Alignment (EMA) for robust image-text alignment and a Cross-Modal Diffusion Reconstruction (CDR) module that diffuses visual information into semantic representations. EMA enforces cross-modal consistency via a contrastive, cosine-based alignment loss, while CDR uses diffusion guidance from visual features to reconstruct semantically rich representations, optimized with MSE and cross-entropy constraints. A fusion stage combines EMA and CDR outputs for final classification, and the approach is validated on Vireo-Food172 and Ingredient-101, where MARNet achieves state-of-the-art performance and demonstrates improved robustness via ablative analyses and case studies. The work highlights the practical value of integrating diffusion-based reconstruction with cross-modal alignment to enhance image understanding in multimodal settings, with future work focused on reducing diffusion-associated noise while maintaining representational quality.

Abstract

Image classification models often demonstrate unstable performance in real-world applications due to variations in image information, driven by differing visual perspectives of subject objects and lighting discrepancies. To mitigate these challenges, existing studies commonly incorporate additional modal information matching the visual data to regularize the model's learning process, enabling the extraction of high-quality visual features from complex image regions. Specifically, in the realm of multimodal learning, cross-modal alignment is recognized as an effective strategy, harmonizing different modal information by learning a domain-consistent latent feature space for visual and semantic features. However, this approach may face limitations due to the heterogeneity between multimodal information, such as differences in feature distribution and structure. To address this issue, we introduce a Multimodal Alignment and Reconstruction Network (MARNet), designed to enhance the model's resistance to visual noise. Importantly, MARNet includes a cross-modal diffusion reconstruction module for smoothly and stably blending information across different domains. Experiments conducted on two benchmark datasets, Vireo-Food172 and Ingredient-101, demonstrate that MARNet effectively improves the quality of image information extracted by the model. It is a plug-and-play framework that can be rapidly integrated into various image classification frameworks, boosting model performance.

Figures (4)

• Figure 1: The proposed MARNet schematic diagram. The network mainly comprises alignment and diffusion modules, wherein the alignment module matches and aligns image and text information, and the diffusion module reconstructs the distribution of image information.
• Figure 2: The framework diagram of MARNet. The input to MARNet is image-text data pairs, which are processed by neural networks for vision and text to obtain $x_v$ and $x_s$, respectively. Modules EMA and CDR handle the multi-modal representations and output representations $x_{EMA}$ and $x_{CDR}$, which are fused in the end.
• Figure 3: Visualization of the basic representation $x_v$ and reconstructed representation $x_{CDR}$ using t-SNE. As shown in the legend, the color of dots represents the category.
• Figure 4: Prediction results of the basic module and CDR module. The minimal confidence values are represented in scientific notation, e.g., 6.4e-1 indicates 0.64.