RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation

TL;DR

RoutIR tackles the challenge of serving retrieval pipelines online in retrieval-augmented generation by wrapping diverse retrieval methods behind a lightweight HTTP API. It introduces three architectural layers—Engines, Processors, and Pipelines—with a Relay Engine for distributed deployment and an on-demand pipeline language for dynamic composition. Empirical evaluation on NeuCLIR and RAG-related tracks shows strong throughput (3 to 10 queries per second) and CPU-only latency around 600 ms, with substantial gains from batching and caching. The work provides an open-source, easily extensible framework that lowers engineering barriers for integrating state-of-the-art retrieval methods into real-time RAG systems.

Abstract

Retrieval models are key components of Retrieval-Augmented Generation (RAG) systems, which generate search queries, process the documents returned, and generate a response. RAG systems are often dynamic and may involve multiple rounds of retrieval. While many state-of-the-art retrieval methods are available through academic IR platforms, these platforms are typically designed for the Cranfield paradigm in which all queries are known up front and can be batch processed offline. This simplification accelerates research but leaves state-of-the-art retrieval models unable to support downstream applications that require online services, such as arbitrary dynamic RAG pipelines that involve looping, feedback, or even self-organizing agents. In this work, we introduce RoutIR, a Python package that provides a simple and efficient HTTP API that wraps arbitrary retrieval methods, including first stage retrieval, reranking, query expansion, and result fusion. By providing a minimal JSON configuration file specifying the retrieval models to serve, RoutIR can be used to construct and query retrieval pipelines on-the-fly using any permutation of available models (e.g., fusing the results of several first-stage retrieval methods followed by reranking). The API automatically performs asynchronous query batching and caches results by default. While many state-of-the-art retrieval methods are already supported by the package, RoutIR is also easily expandable by implementing the Engine abstract class. The package is open-sourced and publicly available on GitHub: http://github.com/hltcoe/routir.

Figures (5)

• Figure 1: Service architecture of RoutIR. The user's HTTP request specifies a retrieval pipeline and search query, illustrated on the left. RoutIR orchestrates the retrieval pipeline on the fly and processes the query with each query processor. Each processor manages queuing, caching, and query processing. The "Next Processor" enables, for example, reranking or result fusion. See Figure \ref{['fig:pipeline-request']} for a full JSON example of a request.
• Figure 2: Example Pipeline Request. The pipeline issues the query to qwen3-neuclir and plaidx-neuclir engines, fuses the results with reciprocal rank fusion, takes the top 50 documents from the fused result, and finally reranks using Rank1 weller2025rank1testtimecomputereranking reranker.
• Figure 3: Example configuration file. Refer to the RoutIR documentation for more fields. In the example, we load two engines: Rank1 reranker from a custom script (see Section \ref{['sec:usage:integration']}), and qwen3-neuclir using a FAISS index loaded from Huggingface Datasets containing Qwen3 8B embeddings. The specified document collection can be used to retrieve document text via an API or to pass documents to reranking models.
• Figure 4: Example code snippet for integrating Pyserini with RoutIR. This example can be extended with more flexible parameter configuration or even allowing the endpoint users to specify the retrieval model. Full example can be found at https://github.com/hltcoe/routir/blob/main/examples/pyserini_extension.py.
• Figure 5: Example search module for GPT Researcher.