Semantic-Space-Intervened Diffusive Alignment for Visual Classification

TL;DR

Cross-modal alignment for visual classification is hampered by semantic and distributional differences between visual and textual features. We introduce Semantic-Space-Intervened Diffusive Alignment (SeDA), a diffusion-based framework that uses a modality-shared semantic space as an intermediary and a bi-stage diffusion process (DSL and DST), complemented by a Progressive Feature Interaction Network (PFIN) for stepwise multimodal fusion. The visual space is first mapped into semantic space and then translated into the textual distribution, with diffusion steps performed over $T$ and a staged transition at step $t$, achieving progressive alignment. Empirical results on VIREO Food-172, NUS-WIDE, and MSRVTT show strong gains over baselines, with ablations validating the contributions of DSL, DST, and PFIN and case studies highlighting improved semantic disambiguation. This diffusion-driven cross-modal paradigm offers a model-agnostic approach that can be integrated with diverse visual backbones and motivates future diffusion-enabled multimodal learning.

Abstract

Cross-modal alignment is an effective approach to improving visual classification. Existing studies typically enforce a one-step mapping that uses deep neural networks to project the visual features to mimic the distribution of textual features. However, they typically face difficulties in finding such a projection due to the two modalities in both the distribution of class-wise samples and the range of their feature values. To address this issue, this paper proposes a novel Semantic-Space-Intervened Diffusive Alignment method, termed SeDA, models a semantic space as a bridge in the visual-to-textual projection, considering both types of features share the same class-level information in classification. More importantly, a bi-stage diffusion framework is developed to enable the progressive alignment between the two modalities. Specifically, SeDA first employs a Diffusion-Controlled Semantic Learner to model the semantic features space of visual features by constraining the interactive features of the diffusion model and the category centers of visual features. In the later stage of SeDA, the Diffusion-Controlled Semantic Translator focuses on learning the distribution of textual features from the semantic space. Meanwhile, the Progressive Feature Interaction Network introduces stepwise feature interactions at each alignment step, progressively integrating textual information into mapped features. Experimental results show that SeDA achieves stronger cross-modal feature alignment, leading to superior performance over existing methods across multiple scenarios.

Figures (5)

• Figure 1: Common feature alignment methods and the proposed SeDA alignment framework. In (a), traditional feature alignment processes fail to capture the underlying distribution of textual features, resulting in persistent inter-class confusion. In (b), SeDA employs a semantic-space-intervened diffusive alignment method, transferring visual features to the textual features space step by step through a bi-stage learning process.
• Figure 2: Illustration of the proposed SeDA. SeDA takes visual-textual data pairs as input, which are processed by dedicated neural networks for vision and text to extract global features $x_v$ and $x_s$. The PFIN module progressively integrates textual information, while the DSL and DST modules work together to align visual and textual features effectively.
• Figure 3: The impact of hyperparameters on performance. The weight parameter $\gamma$, the time step $T$ and the staged step $t$ are turned from {0.5,1.0,1.5,2.0}, {900,1200,1500,1800}, {0,300,500} on VIREO Food-172, respectively.
• Figure 4: T-SNE visualization of data distribution before and after alignment for a randomly selected category.
• Figure 5: Comparison of ViT-B/16 and SeDA on confusion matrix and randomly selected samples with Top-5 confidence scores. Red boxes represent baseline results, green boxes represent SeDA results, and ground-truth labels are highlighted in bold.