Assessing generalization capability of text ranking models in Polish

TL;DR

The paper tackles the generalization problem of Polish text ranking in retrieval-augmented generation by evaluating Polish and multilingual rerankers on the 41-task PIRB benchmark. It demonstrates that most rerankers lag behind dense retrievers in zero-shot settings, but that large-scale optimization and diverse training data can yield compact, generalizable rerankers. Through knowledge distillation from a 13B MT5 teacher using 1.4M queries, a 435M-parameter model achieves $62.65$ on the PIRB $NDCG@10$ metric, about $30×$ fewer parameters and $33×$ faster than the teacher, effectively matching or exceeding the teacher’s performance. The findings provide practical guidance on when to deploy rerankers in Polish RAG systems and show a viable path to efficient, high-performing Polish rerankers that establish new state-of-the-art results.

Abstract

Retrieval-augmented generation (RAG) is becoming an increasingly popular technique for integrating internal knowledge bases with large language models. In a typical RAG pipeline, three models are used, responsible for the retrieval, reranking, and generation stages. In this article, we focus on the reranking problem for the Polish language, examining the performance of rerankers and comparing their results with available retrieval models. We conduct a comprehensive evaluation of existing models and those trained by us, utilizing a benchmark of 41 diverse information retrieval tasks for the Polish language. The results of our experiments show that most models struggle with out-of-domain generalization. However, a combination of effective optimization method and a large training dataset allows for building rerankers that are both compact in size and capable of generalization. The best of our models establishes a new state-of-the-art for reranking in the Polish language, outperforming existing models with up to 30 times more parameters.

Figures (2)

• Figure 1: A diagram showing the three-step RAG workflow. In the first step, the retriever model extracts a list of $k0$ best-matching documents from the document collection based on the user's query. Subsequently, the documents are sorted using the reranker, which returns the top $k$ documents ordered by their relevance to the query, where $k \leq k0$. The query is then passed to the reader model, along with the context that includes the text of the retrieved documents. The model is responsible for generating the final answer using the context in either an extractive or abstractive manner.
• Figure 2: Comparison of NDCG@10 scores between first-stage retrieval and reranking for the top two rerankers, broken down into individual datasets from the PIRB benchmark.