Mistral-SPLADE: LLMs for better Learned Sparse Retrieval

TL;DR

This work addresses improving learned sparse retrieval by leveraging decoder-only LLMs to learn semantic keyword expansions. It introduces Echo-Mistral SPLADE, which combines LoRA-finetuned Mistral with echo embeddings to train a sparse retriever that retains interpretability while benefiting from large-scale pretraining. On BEIR, it achieves state-of-the-art sparse retrieval results and shows competitive performance against dense models in zero-shot settings, demonstrating the practical viability of LSR with LLM backbones. The findings highlight the value of diverse training data, model scale, and memory-efficient tuning for robust, fast retrieval with sparse representations.

Abstract

Learned Sparse Retrievers (LSR) have evolved into an effective retrieval strategy that can bridge the gap between traditional keyword-based sparse retrievers and embedding-based dense retrievers. At its core, learned sparse retrievers try to learn the most important semantic keyword expansions from a query and/or document which can facilitate better retrieval with overlapping keyword expansions. LSR like SPLADE has typically been using encoder only models with MLM (masked language modeling) style objective in conjunction with known ways of retrieval performance improvement such as hard negative mining, distillation, etc. In this work, we propose to use decoder-only model for learning semantic keyword expansion. We posit, decoder only models that have seen much higher magnitudes of data are better equipped to learn keyword expansions needed for improved retrieval. We use Mistral as the backbone to develop our Learned Sparse Retriever similar to SPLADE and train it on a subset of sentence-transformer data which is often used for training text embedding models. Our experiments support the hypothesis that a sparse retrieval model based on decoder only large language model (LLM) surpasses the performance of existing LSR systems, including SPLADE and all its variants. The LLM based model (Echo-Mistral-SPLADE) now stands as a state-of-the-art learned sparse retrieval model on the BEIR text retrieval benchmark.

Figures (1)

• Figure 1: Learned Sparse Retrieval: Training vs Inference