Semantic-Cohesive Knowledge Distillation for Deep Cross-modal Hashing

TL;DR

The paper tackles cross-modal hashing for image-text retrieval by identifying a misalignment between self-supervised multi-label semantics and heterogeneous modalities. It introduces SODA, a semantic cohesive knowledge distillation framework that uses ground-truth label prompts as a textual modality and a two-stage teacher-student network to align image and label spaces before supervising the text modality. The approach yields a well-mapped $L$-bit Hamming space, demonstrated to outperform state-of-the-art baselines on MIRFLICKR-25K and NUS-WIDE across multiple hash lengths, with robustness confirmed via cross-validation and ablation analyses. The work enables more accurate and scalable cross-modal retrieval by tightly coupling semantic supervision with explicit cross-modal interactions, leveraging CLIP encoders and prompt-based label representations.

Abstract

Recently, deep supervised cross-modal hashing methods have achieve compelling success by learning semantic information in a self-supervised way. However, they still suffer from the key limitation that the multi-label semantic extraction process fail to explicitly interact with raw multimodal data, making the learned representation-level semantic information not compatible with the heterogeneous multimodal data and hindering the performance of bridging modality gap. To address this limitation, in this paper, we propose a novel semantic cohesive knowledge distillation scheme for deep cross-modal hashing, dubbed as SODA. Specifically, the multi-label information is introduced as a new textual modality and reformulated as a set of ground-truth label prompt, depicting the semantics presented in the image like the text modality. Then, a cross-modal teacher network is devised to effectively distill cross-modal semantic characteristics between image and label modalities and thus learn a well-mapped Hamming space for image modality. In a sense, such Hamming space can be regarded as a kind of prior knowledge to guide the learning of cross-modal student network and comprehensively preserve the semantic similarities between image and text modality. Extensive experiments on two benchmark datasets demonstrate the superiority of our model over the state-of-the-art methods.

Figures (6)

• Figure 1: Illustration of two model optimization strategies of existing cross-modal hashing methods.
• Figure 2: Illustration of the proposed SODA, where the cross-modal teacher network is first designed to distill knowledge of image modality by directly narrowing the image-label modality gap and the cross-modal student network is then trained using the distilled image modality knowledge.
• Figure 3: The P-R curves of different methods on two datasets, where CLIP features are utilized for baseline methods and the hash code length is $64$ bits.
• Figure 4: The five-fold cross-validation results of SODA and baseline methods on two datasets and the code length is set as $64$.
• Figure 5: Sensitivity analysis of the hyper-parameters.