Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching

TL;DR

The paper tackles image-text matching by addressing insufficient distance separability in triplet learning. It proposes Deep Boosting Learning (DBL), where an anchor branch first learns data- and model-driven distance priors and a target branch uses adaptive margins to enlarge separation between positive and negative pairs, implemented via Relative Sum $\ell'_{RS}$, Relative Max $\ell'_{RM}$, Absolute Sum $\ell'_{AS}$, and Absolute Max $\ell'_{AM}$ losses, with Soft Adaptation $\gamma^{SA}$. DBL supports three training regimes—Offline Asynchronous, Online Synchronous, and Momentum Synchronous—allowing flexible integration with existing cooperative strategies. Experiments on Flickr30K and MSCOCO show consistent improvements across diverse backbones (embedding-based and interaction-based ITM) and demonstrate DBL’s robustness and broad applicability, including extension to larger peer cohorts. The work provides a practical boosting paradigm that enhances cross-modal retrieval without increasing inference cost, and code is publicly available.

Abstract

Image-text matching remains a challenging task due to heterogeneous semantic diversity across modalities and insufficient distance separability within triplets. Different from previous approaches focusing on enhancing multi-modal representations or exploiting cross-modal correspondence for more accurate retrieval, in this paper we aim to leverage the knowledge transfer between peer branches in a boosting manner to seek a more powerful matching model. Specifically, we propose a brand-new Deep Boosting Learning (DBL) algorithm, where an anchor branch is first trained to provide insights into the data properties, with a target branch gaining more advanced knowledge to develop optimal features and distance metrics. Concretely, an anchor branch initially learns the absolute or relative distance between positive and negative pairs, providing a foundational understanding of the particular network and data distribution. Building upon this knowledge, a target branch is concurrently tasked with more adaptive margin constraints to further enlarge the relative distance between matched and unmatched samples. Extensive experiments validate that our DBL can achieve impressive and consistent improvements based on various recent state-of-the-art models in the image-text matching field, and outperform related popular cooperative strategies, e.g., Conventional Distillation, Mutual Learning, and Contrastive Learning. Beyond the above, we confirm that DBL can be seamlessly integrated into their training scenarios and achieve superior performance under the same computational costs, demonstrating the flexibility and broad applicability of our proposed method. Our code is publicly available at: https://github.com/Paranioar/DBL.

Figures (9)

• Figure 1: Illustration of our single branch baseline. We adopt hard ranking loss [14] as a task-specific loss to supervise the training process.
• Figure 2: Illustration of deep boosting learning. We first perform the anchor branch to obtain the absolute distance ($\mathcal{S}_{\boldsymbol{q}, \boldsymbol{p}}^{a},\mathcal{S}_{\boldsymbol{q}, \boldsymbol{n_{i}}}^{a}|_{i=1,2}$) between query and each candidate, and relative distance ($\mathcal{S}_{\boldsymbol{q}, \boldsymbol{p}}^{a} - \mathcal{S}_{\boldsymbol{q}, {\boldsymbol{n_{i}}}}^{a}|_{i=1,2}$) within each triplet. Based on this prior knowledge, we assign the target branch with appropriate thresholds to further enlarge the variations between matched and unmatched image-text pairs.
• Figure 3: Illustration of multiple training scenarios. OAS adopts two-stage training scheme as with Conventional Distillation, while OSS and MSS employ one-stage parallel training scenario as with Mutual Learning and Contrastive Learning, respectively. Notably, we only verify the target branch on the validation set, and utilize the model with the best RSUM to perform prediction on the test set.
• Figure 4: Analyses of hyperparameter $\gamma$ for RM and $\alpha$ for AM under OSS.
• Figure 5: Comparison of feature distribution for image and text samples. We implement t-SNE to visualize the image and text features based on GPO ITM:GPO.