Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

Thong Nguyen, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

TL;DR

The paper tackles multimodal retrieval by turning dense image-caption representations into sparse lexical vectors via a lightweight projection head, enabling efficient inverted-index retrieval. It introduces Dense2Sparse, a method that freezes dense encoders (e.g., BLIP/ALBEF) and trains a sparse projection head with probabilistic expansion control using Bernoulli variables to curb dimension co-activation and semantic deviation. The proposed loss jointly optimizes dense-to-dense alignment and sparse regularization, and experiments on MSCOCO and Flickr30k show that carefully scheduled expansion yields sparse models that are competitive with, and sometimes surpass, existing multimodal LSR systems while reducing training time and memory. The work demonstrates faithful correspondence to dense models and suggests a practical, efficient pathway for deploying multimodal LSR via inverted-index pipelines.

Abstract

Learned sparse retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors that can be indexed and retrieved efficiently with an inverted index. We explore the application of LSR to the multi-modal domain, with a focus on text-image retrieval. While LSR has seen success in text retrieval, its application in multimodal retrieval remains underexplored. Current approaches like LexLIP and STAIR require complex multi-step training on massive datasets. Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors. We address issues of high dimension co-activation and semantic deviation through a new training algorithm, using Bernoulli random variables to control query expansion. Experiments with two dense models (BLIP, ALBEF) and two datasets (MSCOCO, Flickr30k) show that our proposed algorithm effectively reduces co-activation and semantic deviation. Our best-performing sparsified model outperforms state-of-the-art text-image LSR models with a shorter training time and lower GPU memory requirements. Our approach offers an effective solution for training LSR retrieval models in multimodal settings. Our code and model checkpoints are available at github.com/thongnt99/lsr-multimodal

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

TL;DR

The paper tackles multimodal retrieval by turning dense image-caption representations into sparse lexical vectors via a lightweight projection head, enabling efficient inverted-index retrieval. It introduces Dense2Sparse, a method that freezes dense encoders (e.g., BLIP/ALBEF) and trains a sparse projection head with probabilistic expansion control using Bernoulli variables to curb dimension co-activation and semantic deviation. The proposed loss jointly optimizes dense-to-dense alignment and sparse regularization, and experiments on MSCOCO and Flickr30k show that carefully scheduled expansion yields sparse models that are competitive with, and sometimes surpass, existing multimodal LSR systems while reducing training time and memory. The work demonstrates faithful correspondence to dense models and suggests a practical, efficient pathway for deploying multimodal LSR via inverted-index pipelines.

Abstract

Learned sparse retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors that can be indexed and retrieved efficiently with an inverted index. We explore the application of LSR to the multi-modal domain, with a focus on text-image retrieval. While LSR has seen success in text retrieval, its application in multimodal retrieval remains underexplored. Current approaches like LexLIP and STAIR require complex multi-step training on massive datasets. Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors. We address issues of high dimension co-activation and semantic deviation through a new training algorithm, using Bernoulli random variables to control query expansion. Experiments with two dense models (BLIP, ALBEF) and two datasets (MSCOCO, Flickr30k) show that our proposed algorithm effectively reduces co-activation and semantic deviation. Our best-performing sparsified model outperforms state-of-the-art text-image LSR models with a shorter training time and lower GPU memory requirements. Our approach offers an effective solution for training LSR retrieval models in multimodal settings. Our code and model checkpoints are available at github.com/thongnt99/lsr-multimodal
Paper Structure (11 sections, 7 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 11 sections, 7 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Methodology
  5. Model architecture
  6. Training loss
  7. Experiments and Results
  8. Experimental setup
  9. Results and discussion
  10. Retrieval latency of dense and sparsified models
  11. Conclusion

Figures (2)

  • Figure 1: The architecture of Dense2Sparse (D2S). The caption and image encoders are frozen, and the sparse projection is trained to project dense vectors to sparse vectors.
  • Figure 2: Sparisified models compared to original dense models.

Theorems & Definitions (2)

  • definition thmcounterdefinition: Dimension co-activation
  • definition thmcounterdefinition: Semantic deviation