Effective Inference-Free Retrieval for Learned Sparse Representations

TL;DR

The paper investigates how regularization governs term expansion in Learned Sparse Retrieval (LSR) and identifies a query-encoding bottleneck when using inverted-index engines. It shows that relaxing regularization can improve retrieval effectiveness and that modern engines can handle larger expansions efficiently. To address the query-encoding bottleneck, it introduces Learned Inference-free Retrieval (Li-LSR), which learns a per-token score during training and replaces encoding with a fast dictionary lookup, achieving state-of-the-art results on both in-domain (MsMarco) and out-of-domain (BEIR) benchmarks. Li-LSR outperforms existing inference-free methods, even with simpler training setups, and combining learned scores with IDF can boost performance for smaller models. The work points to practical gains in speed and accuracy for sparse representations and suggests compression of learned sparse representations as future work.

Abstract

Learned Sparse Retrieval (LSR) is an effective IR approach that exploits pre-trained language models for encoding text into a learned bag of words. Several efforts in the literature have shown that sparsity is key to enabling a good trade-off between the efficiency and effectiveness of the query processor. To induce the right degree of sparsity, researchers typically use regularization techniques when training LSR models. Recently, new efficient -- inverted index-based -- retrieval engines have been proposed, leading to a natural question: has the role of regularization changed in training LSR models? In this paper, we conduct an extended evaluation of regularization approaches for LSR where we discuss their effectiveness, efficiency, and out-of-domain generalization capabilities. We first show that regularization can be relaxed to produce more effective LSR encoders. We also show that query encoding is now the bottleneck limiting the overall query processor performance. To remove this bottleneck, we advance the state-of-the-art of inference-free LSR by proposing Learned Inference-free Retrieval (Li-LSR). At training time, Li-LSR learns a score for each token, casting the query encoding step into a seamless table lookup. Our approach yields state-of-the-art effectiveness for both in-domain and out-of-domain evaluation, surpassing Splade-v3-Doc by 1 point of mRR@10 on MS MARCO and 1.8 points of nDCG@10 on BEIR.

Figures (1)

• Figure 1: Efficiency-effectiveness trade-off (AQT vs. mRR@10) of different LSR models with different expansion.