DeeperImpact: Optimizing Sparse Learned Index Structures

TL;DR

DeeperImpact addresses the gap between sparse learned index retrieval and dense or heavily optimized sparse methods by revisiting and upgrading DeepImpact. It combines Llama 2-based document expansion with CoCondenser initialization, hard negatives, and distillation to boost effectiveness while maintaining fast query processing. The results show substantial improvements over the original DeepImpact and competitive performance relative to SPLADE, particularly in recall and ranking precision, with significantly lower latency than the strongest SPLADE variants. These findings suggest DeeperImpact offers a practical, scalable option for large-scale IR systems seeking strong effectiveness without sacrificing efficiency.

Abstract

A lot of recent work has focused on sparse learned indexes that use deep neural architectures to significantly improve retrieval quality while keeping the efficiency benefits of the inverted index. While such sparse learned structures achieve effectiveness far beyond those of traditional inverted index-based rankers, there is still a gap in effectiveness to the best dense retrievers, or even to sparse methods that leverage more expensive optimizations such as query expansion and query term weighting. We focus on narrowing this gap by revisiting and optimizing DeepImpact, a sparse retrieval approach that uses DocT5Query for document expansion followed by a BERT language model to learn impact scores for document terms. We first reinvestigate the expansion process and find that the recently proposed Doc2Query -- query filtration does not enhance retrieval quality when used with DeepImpact. Instead, substituting T5 with a fine-tuned Llama 2 model for query prediction results in a considerable improvement. Subsequently, we study training strategies that have proven effective for other models, in particular the use of hard negatives, distillation, and pre-trained CoCondenser model initialization. Our results substantially narrow the effectiveness gap with the most effective versions of SPLADE.