GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning

TL;DR

A Generalized Structural Sparse Function is proposed to dynamically capture thorough and powerful relationships across modalities for pair-wise similarity learning while remaining concise but efficient and reaches a sweet spot between model complexity and capability.

Abstract

Cross-modal metric learning is a prominent research topic that bridges the semantic heterogeneity between vision and language. Existing methods frequently utilize simple cosine or complex distance metrics to transform the pairwise features into a similarity score, which suffers from an inadequate or inefficient capability for distance measurements. Consequently, we propose a Generalized Structural Sparse Function to dynamically capture thorough and powerful relationships across modalities for pair-wise similarity learning while remaining concise but efficient. Specifically, the distance metric delicately encapsulates two formats of diagonal and block-diagonal terms, automatically distinguishing and highlighting the cross-channel relevancy and dependency inside a structured and organized topology. Hence, it thereby empowers itself to adapt to the optimal matching patterns between the paired features and reaches a sweet spot between model complexity and capability. Extensive experiments on cross-modal and two extra uni-modal retrieval tasks (image-text retrieval, person re-identification, fine-grained image retrieval) have validated its superiority and flexibility over various popular retrieval frameworks. More importantly, we further discover that it can be seamlessly incorporated into multiple application scenarios, and demonstrates promising prospects from Attention Mechanism to Knowledge Distillation in a plug-and-play manner. Our code is publicly available at: https://github.com/Paranioar/GSSF.

Figures (9)

• Figure 1: Illustration of the proposed GSSF metrics. The gray and orange parts indicate the fixed and trainable matrix parameters respectively during the training process. Compared with Cosine and Dense metrics, the Diag item seeks distinct and appropriate weights for within-channel connections, while the B-Diag item further constructs between-channel connectivity and sparsity by structural topology. Both of them are structural sparse strategies of Dense metric.
• Figure 2: Illustration of several applications. For (a) Attention Mechanism, it adaptively refines the interactions between query and key from mono- or cross-modality features, while for (b) Fine-grained Alignments, it enhances pairwise similarity scores based on various popular token-wise integration. Besides, we validate its potential by calibrating the teacher's network outputs and adaptation representations during (c) Knowledge Distillation and (d) Transfer Learning.
• Figure 3: Configuration of various block sizes $d$. The embedding-based VSRN and interaction-based SCAN are utilized to analyze the influence of the ratio $N$ ($D/d$) between the feature dimension $D$ and block size $d$. The smaller the block size $d$, the sparser the channel-wise connection $\boldsymbol{W}$.
• Figure 4: Visualization of cross-channel weight coefficients learned by the B-Diag metric. For clear visualization, we reduce the size of the matrix $\boldsymbol{W}$ to 64x64 at regular intervals. Note that red and blue zones indicate positive and negative correlations and relationships across channels, respectively.
• Figure 5: Visualization of the learned similarity distributions of all positive and negative image-text pairs with Cosine/Diag/B-Diag metric on Flickr30K.