To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval

TL;DR

This paper addresses how the casing of backbone models affects Learned Sparse Retrieval (LSR) by conducting a systematic, cross-dataset evaluation of paired cased and uncased encoders across MSMARCO, DL-2019, DL-2020, and BEIR. It introduces controlled pre-processing (lowercasing) and post-processing (uncased-vocabulary logits and cased regularizer) within a Margin-MSE distillation framework with FLOPs-based sparsity to study vocabulary-space effects. The key findings show that uncased backbones outperform without pre-processing, but applying lowercasing recovers cased-model performance to near the uncased baseline, while post-processing yields efficiency gains with only modest accuracy costs; token-level analysis explains the underlying mechanism by showing cased tokens are largely suppressed under lowercasing. Significantly, the results broaden the viability of using stronger cased architectures in LSR, enabling easier integration with modern backbones, and the authors provide publicly available code for reproducibility $MRR@10$ and $nDCG@10$ evaluations across datasets.

Abstract

Learned Sparse Retrieval (LSR) methods construct sparse lexical representations of queries and documents that can be efficiently searched using inverted indexes. Existing LSR approaches have relied almost exclusively on uncased backbone models, whose vocabularies exclude case-sensitive distinctions, thereby reducing vocabulary mismatch. However, the most recent state-of-the-art language models are only available in cased versions. Despite this shift, the impact of backbone model casing on LSR has not been studied, potentially posing a risk to the viability of the method going forward. To fill this gap, we systematically evaluate paired cased and uncased versions of the same backbone models across multiple datasets to assess their suitability for LSR. Our findings show that LSR models with cased backbone models by default perform substantially worse than their uncased counterparts; however, this gap can be eliminated by pre-processing the text to lowercase. Moreover, our token-level analysis reveals that, under lowercasing, cased models almost entirely suppress cased vocabulary items and behave effectively as uncased models, explaining their restored performance. This result broadens the applicability of recent cased models to the LSR setting and facilitates the integration of stronger backbone architectures into sparse retrieval. The complete code and implementation for this project are available at: https://github.com/lionisakis/Uncased-vs-cased-models-in-LSR

Figures (2)

• Figure 1: Pipeline of cased models. Queries and documents first undergo a pre-processing step, followed by encoding, and then a post-processing step where sparse vectors are generated and compared. During post-processing, Cased Regularization is applied only during training as an additional loss.
• Figure 2: Confusion matrices comparing input and output token casing across BERT and DistilBERT models under different pre-processing conditions. For both models, no post-processing method is used. Rows correspond to input token casing (cased vs. uncased), and columns represent the resulting output token casing. Values denote absolute token counts rather than normalized proportions, highlighting the relative dominance of uncased tokens across conditions.