MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model

TL;DR

This work proposes an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems and exhibits a significant performance enhancement, outperforming previous approaches and achieving state-of-the-art results on ReQA SQuAD.

Abstract

Large Language Models (LLMs) often struggle with hallucinations and outdated information. To address this, Information Retrieval (IR) systems can be employed to augment LLMs with up-to-date knowledge. However, existing IR techniques contain deficiencies, posing a performance bottleneck. Given the extensive array of IR systems, combining diverse approaches presents a viable strategy. Nevertheless, prior attempts have yielded restricted efficacy. In this work, we propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems. We demonstrate the method on two Retrieval Question Answering (ReQA) tasks. Our empirical findings exhibit a significant performance enhancement, outperforming previous approaches and achieving state-of-the-art results on ReQA SQuAD.

Figures (4)

• Figure 1: Our framework for amalgamating multiple ranking information by using the re-ranking approach.
• Figure 2: Data preparation steps include: 1) retrieving scores for each document candidate, 2) performing permutations on each pair of documents and filtering out tied pairs, and 3) creating negative and positive samples.
• Figure 3: Relationship between the number of re-ranked documents ($k$) and retrieval performance (MRR).
• Figure 4: Relationship between the number of re-ranked documents ($k$) and inference time (query/sec).