Making Large Language Models Efficient Dense Retrievers

TL;DR

The paper tackles the high computational cost of LLM-based dense retrievers by revealing that MLPs are the primary source of redundancy for retrieval tasks, unlike generation where attention often dominates pruning. It introduces EffiR, a two-stage coarse-to-fine compression strategy that first drops entire MLP layers and then self-slims the remaining MLP widths, followed by retrieval-specific fine-tuning. Across BEIR and multiple backbones, EffiR achieves substantial parameter reductions and near-full performance, with significant query-speedups and compatibility with quantization. This provides a practical pathway to deploy efficient, high-quality dense retrievers in real-world settings.

Abstract

Recent work has shown that directly fine-tuning large language models (LLMs) for dense retrieval yields strong performance, but their substantial parameter counts make them computationally inefficient. While prior studies have revealed significant layer redundancy in LLMs for generative tasks, it remains unclear whether similar redundancy exists when these models are adapted for retrieval tasks, which require encoding entire sequences into fixed representations rather than generating tokens iteratively. To this end, we conduct a comprehensive analysis of layer redundancy in LLM-based dense retrievers. We find that, in contrast to generative settings, MLP layers are substantially more prunable, while attention layers remain critical for semantic aggregation. Building on this insight, we propose EffiR, a framework for developing efficient retrievers that performs large-scale MLP compression through a coarse-to-fine strategy (coarse-grained depth reduction followed by fine-grained width reduction), combined with retrieval-specific fine-tuning. Across diverse BEIR datasets and LLM backbones, EffiR achieves substantial reductions in model size and inference cost while preserving the performance of full-size models.

Figures (7)

• Figure 1: Effectiveness–efficiency trade-off in LLM-based dense retrievers. Each point shows a model’s BEIR performance vs. inference speed. All models are fine-tuned on MS MARCO. Marker types indicate compression strategies. EffiR builds efficient models based on Mistral-7B. For example, EffiR-20M-3.6B applies our EffiR method by dropping 20 MLP layers from Mistral-7B, then fine-tuning the remaining 3.6B parameters. MLP-pruned models (green squares) consistently lie near the Pareto frontier (dashed), showing strong efficiency with minimal accuracy loss.
• Figure 2: Analyzing compression behavior across layers.
• Figure 3: Comparison of coarse-grained layer dropping and fine-grained self-slimming, both starting from the same 16-layer dropped base Mistral-7B model.
• Figure 4: Comparison of coarse-grained layer dropping and fine-grained self-slimming, both starting from the same 16-layer dropped base Qwen2.5-7B model.
• Figure 5: Dropping‑order heat maps for the base model (Mistral‑7B) and its embedding variant (E5‑Mistral).