Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval

TL;DR

Lightning IR addresses the complexity of applying transformer-based models in information retrieval by delivering a backbone-agnostic, end-to-end PyTorch Lightning framework for fine-tuning, indexing, searching, and re-ranking. It unifies diverse model types (bi- and cross-encoders) and data sources within modular components (model, dataset, trainer, CLI), supporting a range of architectures and retrieval configurations. A reproducibility study on TREC DL benchmarks demonstrates competitive $nDCG@10$ performance and confirms that state-of-the-art models can be reproduced with minimal effort using Lightning IR and HF configurations. The framework offers practical tooling, including a CLI and ir_datasets integration, and outlines future work to expand efficient indexing pipelines and scalability for multi-vector and sparse retrieval.

Abstract

A wide range of transformer-based language models have been proposed for information retrieval tasks. However, including transformer-based models in retrieval pipelines is often complex and requires substantial engineering effort. In this paper, we introduce Lightning IR, an easy-to-use PyTorch Lightning-based framework for applying transformer-based language models in retrieval scenarios. Lightning IR provides a modular and extensible architecture that supports all stages of a retrieval pipeline: from fine-tuning and indexing to searching and re-ranking. Designed to be scalable and reproducible, Lightning IR is available as open-source: https://github.com/webis-de/lightning-ir.