Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking

TL;DR

This work addresses the gap in contrastive learning methods that ignore fine-grained ranking signals, proposing Generalized Contrastive Learning (GCL) to fuse relevance and ranking into a single-stage, multi-modal retrieval framework. GCL learns from continuous ranking scores by mapping them to per-pair weights, applies a weighted cross-entropy loss, and extends to multi-field representations, enabling unified training across text and image modalities. The authors introduce MarqoGS-10M, a large public dataset with 10 million ranked query–document pairs derived from Google Shopping listings, and demonstrate substantial in-domain and cold-start improvements in $NDCG$ and $ERR$ over CLIP baselines, along with strong offline ecommerce results. The work highlights the potential of ranking-aware embeddings for ranked retrieval and opens avenues for further improvements in recall at high ranks and broader dataset benchmarks.

Abstract

Contrastive learning has gained widespread adoption for retrieval tasks due to its minimal requirement for manual annotations. However, popular training frameworks typically learn from binary (positive/negative) relevance, making them ineffective at incorporating desired rankings. As a result, the poor ranking performance of these models forces systems to employ a re-ranker, which increases complexity, maintenance effort and inference time. To address this, we introduce Generalized Contrastive Learning (GCL), a training framework designed to learn from continuous ranking scores beyond binary relevance. GCL encodes both relevance and ranking information into a unified embedding space by applying ranking scores to the loss function. This enables a single-stage retrieval system. In addition, during our research, we identified a lack of public multi-modal datasets that benchmark both retrieval and ranking capabilities. To facilitate this and future research for ranked retrieval, we curated a large-scale MarqoGS-10M dataset using GPT-4 and Google Shopping, providing ranking scores for each of the 10 million query-document pairs. Our results show that GCL achieves a 29.3% increase in NDCG@10 for in-domain evaluations and 6.0% to 10.0% increases for cold-start evaluations compared to the finetuned CLIP baseline with MarqoGS-10M. Additionally, we evaluated GCL offline on a proprietary user interaction data. GCL shows an 11.2% gain for in-domain evaluations. The dataset and the method are available at: https://github.com/marqo-ai/GCL.

Figures (4)

• Figure 1: Overview of the Generalized Contrastive Learning. GCL integrates ranking information alongside multiple input fields for each sample across both left-hand-side (LHS) and right-hand-side (RHS). Ground-truth ranking scores are transformed into weights, which are used for computing contrastive losses, ensuring that pairs with higher weights incur greater penalties.
• Figure 2: Plot of various score-to-weight functions.
• Figure 3: Overview of the MarqoGS-10M data curation and Quadruple Split. We first extract 2.4k leaf categories from Amazon fashion and homeware sections, which we use to prompt GPT-4 for query generation. Each query is used to retrieve 100 relevant documents via the Google Shopping API, with their listing positions converted to ranking scores, culminating in 10 million triplets. Finally, the data is split in a quadruple way that reflects real world search system.
• Figure 4: NDCG vs k over proprietary in-domain data.