Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Challenges and Opportunities of Learned Sparse Retrieval for Code

Simon Lupart, Maxime Louis, Thibault Formal, Hervé Déjean, Stéphane Clinchant

Abstract

Retrieval over large codebases is a key component of modern LLM-based software engineering systems. Existing approaches predominantly rely on dense embedding models, while learned sparse retrieval (LSR) remains largely unexplored for code. However, applying sparse retrieval to code is challenging due to subword fragmentation, semantic gaps between natural-language queries and code, diversity of programming languages and sub-tasks, and the length of code documents, which can harm sparsity and latency. We introduce SPLADE-Code, the first large-scale family of learned sparse retrieval models specialized for code retrieval (600M-8B parameters). Despite a lightweight one-stage training pipeline, SPLADE-Code achieves state-of-the-art performance among retrievers under 1B parameters (75.4 on MTEB Code) and competitive results at larger scales (79.0 with 8B). We show that learned expansion tokens are critical to bridge lexical and semantic matching, and provide a latency analysis showing that LSR enables sub-millisecond retrieval on a 1M-passage collection with little effectiveness loss.

On the Challenges and Opportunities of Learned Sparse Retrieval for Code

Abstract

Retrieval over large codebases is a key component of modern LLM-based software engineering systems. Existing approaches predominantly rely on dense embedding models, while learned sparse retrieval (LSR) remains largely unexplored for code. However, applying sparse retrieval to code is challenging due to subword fragmentation, semantic gaps between natural-language queries and code, diversity of programming languages and sub-tasks, and the length of code documents, which can harm sparsity and latency. We introduce SPLADE-Code, the first large-scale family of learned sparse retrieval models specialized for code retrieval (600M-8B parameters). Despite a lightweight one-stage training pipeline, SPLADE-Code achieves state-of-the-art performance among retrievers under 1B parameters (75.4 on MTEB Code) and competitive results at larger scales (79.0 with 8B). We show that learned expansion tokens are critical to bridge lexical and semantic matching, and provide a latency analysis showing that LSR enables sub-millisecond retrieval on a 1M-passage collection with little effectiveness loss.
Paper Structure (16 sections, 3 equations, 6 figures, 12 tables)

This paper contains 16 sections, 3 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Learned Sparse Retrieval
  5. SPLADE for Code Retrieval
  6. Model Configuration
  7. Experiments
  8. Benchmarks and Evaluation
  9. Main Results
  10. Analysis
  11. Conclusion
  12. Limitations
  13. Hyper-parameters
  14. Datasets Statistics
  15. Additional Analysis
  16. ...and 1 more sections

Figures (6)

  • Figure 1:
  • Figure 2: (Top) Performance in nDCG@10 of SPLADE-Code and dense models on code retrieval benchmark. (Bottom) Example from CodeFeedback-ST. Only a few vocabulary dimensions receive non-zero weights, covering both semantic terms (e.g., Fibonacci, Python, recursive) and lexical tokens (e.g., input, f).
  • Figure 3: SPLADE-Code is a bi-encoder model for retrieval across programming languages (e.g. Text2Code, Code2Code). When queries and documents are in different programming languages, SPLADE-Code learn some shared dimensions in the representation space.
  • Figure 4: Effectiveness--efficiency trade-off on the CodeSearchNet 1M passage collection. The figure reports retrieval latency and effectiveness of SPLADE-Code-8B compared to the Dense-8B model. $(k_q, k_d)$ indicates the pruning used on queries and documents for LSR.
  • Figure 5: Examples of sparse vocabulary activations produced by SPLADE-Code-8B for different inputs. Highest-weighted expansion tokens in the representations capturing query intent, or algorithmic concepts highlighted.
  • ...and 1 more figures