Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG

TL;DR

This work tackles the problem of improving passage ranking within Retrieval-Augmented Generation by evaluating publicly available ranking models using commercially licensable embedding models on Q&A tasks. It introduces NV-RerankQA-Mistral-4B-v3, a cross-encoder reranker derived from pruned Mistral-7B with bi-directional attention and trained via list-wise contrastive learning, achieving up to a ~14% uplift in NDCG@10 over competing rerankers. An extensive ablation study analyzes the impact of model size, attention direction, and loss function, revealing that larger models and bi-directional attention paired with InfoNCE loss yield the best performance. The paper also discusses deployment trade-offs, showing how two-stage pipelines can significantly reduce indexing throughput and detailing latency considerations on GPUs with TensorRT/Triton, providing actionable guidance for real-world RAG systems in industry contexts.

Abstract

Ranking models play a crucial role in enhancing overall accuracy of text retrieval systems. These multi-stage systems typically utilize either dense embedding models or sparse lexical indices to retrieve relevant passages based on a given query, followed by ranking models that refine the ordering of the candidate passages by its relevance to the query. This paper benchmarks various publicly available ranking models and examines their impact on ranking accuracy. We focus on text retrieval for question-answering tasks, a common use case for Retrieval-Augmented Generation systems. Our evaluation benchmarks include models some of which are commercially viable for industrial applications. We introduce a state-of-the-art ranking model, NV-RerankQA-Mistral-4B-v3, which achieves a significant accuracy increase of ~14% compared to pipelines with other rerankers. We also provide an ablation study comparing the fine-tuning of ranking models with different sizes, losses and self-attention mechanisms. Finally, we discuss challenges of text retrieval pipelines with ranking models in real-world industry applications, in particular the trade-offs among model size, ranking accuracy and system requirements like indexing and serving latency / throughput.

Figures (2)

• Figure 1: Illustration of the typical indexing and querying pipelines for multi-stage text retrieval
• Figure 2: Architecture of NV-RerankQA-Mistral-4B-v3 cross-encoder, pruned and adapted from Mistral 7B