Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question Answering

Yiming Zhang, Siyue Zhang, Junbo Zhao, Chen Zhao

TL;DR

The paper tackles long-tail question answering by challenging the notion that dense retrievers underperform on rare entities. It proposes RPDR, a three-stage data augmentation framework that selects easy-to-learn synthetic QA pairs via round-trip prediction using an inverse embedding model (vec2text) and trains a dense retriever on the filtered data. On PopQA and EntityQuestions, RPDR delivers substantial gains in long-tail retrieval and improves end-to-end QA accuracy, even surpassing BM25 under certain conditions; a routing mechanism further combines RPDR with BM25 to capture complementary strengths. The work highlights the critical role of data quality in augmentation for long-tail QA and lays groundwork for routing-based retrieval systems to further boost performance in real-world scenarios.

Abstract

Long-tail question answering presents significant challenges for large language models (LLMs) due to their limited ability to acquire and accurately recall less common knowledge. Retrieval-augmented generation (RAG) systems have shown great promise in mitigating this limitation by integrating external retrieval mechanisms. However, dense retrieval models often face the same difficulties when generalizing to rare or niche knowledge. In this study, we introduce RPDR, a novel data augmentation framework that selects high-quality easy-to-learn training data, to enhance dense retrievers. Our approach is built around three core components: synthetic data generation, data selection with Round-Trip prediction to identify easy-to-learn instances, and retriever training with these instances. We evaluate RPDR on two long-tail retrieval benchmarks, PopQA and EntityQuestion, demonstrating substantial improvements over existing retrievers like BM25 and Contriver, especially on extremely long-tail categories. We identify the strengths and limitations of RPDR through detailed human analysis and propose a dynamic routing mechanism to dynamically route queries to specialized retrieval modules to further improve retrieval performance.

RPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question Answering

TL;DR

The paper tackles long-tail question answering by challenging the notion that dense retrievers underperform on rare entities. It proposes RPDR, a three-stage data augmentation framework that selects easy-to-learn synthetic QA pairs via round-trip prediction using an inverse embedding model (vec2text) and trains a dense retriever on the filtered data. On PopQA and EntityQuestions, RPDR delivers substantial gains in long-tail retrieval and improves end-to-end QA accuracy, even surpassing BM25 under certain conditions; a routing mechanism further combines RPDR with BM25 to capture complementary strengths. The work highlights the critical role of data quality in augmentation for long-tail QA and lays groundwork for routing-based retrieval systems to further boost performance in real-world scenarios.

Abstract

Long-tail question answering presents significant challenges for large language models (LLMs) due to their limited ability to acquire and accurately recall less common knowledge. Retrieval-augmented generation (RAG) systems have shown great promise in mitigating this limitation by integrating external retrieval mechanisms. However, dense retrieval models often face the same difficulties when generalizing to rare or niche knowledge. In this study, we introduce RPDR, a novel data augmentation framework that selects high-quality easy-to-learn training data, to enhance dense retrievers. Our approach is built around three core components: synthetic data generation, data selection with Round-Trip prediction to identify easy-to-learn instances, and retriever training with these instances. We evaluate RPDR on two long-tail retrieval benchmarks, PopQA and EntityQuestion, demonstrating substantial improvements over existing retrievers like BM25 and Contriver, especially on extremely long-tail categories. We identify the strengths and limitations of RPDR through detailed human analysis and propose a dynamic routing mechanism to dynamically route queries to specialized retrieval modules to further improve retrieval performance.
Paper Structure (46 sections, 9 equations, 4 figures, 10 tables)

This paper contains 46 sections, 9 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Long-Tail Question Answering
  4. Retrieval-Augmented Generation (RAG)
  5. Passage retrieval.
  6. Answer generation.
  7. Dataset
  8. Method
  9. Synthetic Data Generation
  10. Training an Inverse Model
  11. Data Selection with Round-trip Prediction
  12. Experiment
  13. Experimental Setup
  14. Baselines.
  15. Metrics.
  16. ...and 31 more sections

Figures (4)

  • Figure 1: (Top) Negative feedback loop in long-tail question answering in large language model-based systems. (Bottom) We challenge existing findings that dense retrievers struggle on long-tail questions, and argue that through appropriate training, dense retrieval based methods RPDR can surpass BM25 on long-tail retrieval.
  • Figure 2: The RPDR framework consists of three main stages: (1) Synthetic Data Generation that generates a pool of long-tail QA pairs. (2) Data Selection with Round-trip Prediction that trains an inverse model to select easy-to-learn samples with reversibility. (3) Retriever Training that trains a dense retriever using the augmented samples.
  • Figure 3: Case study of "John XIX". "Original" means the top-1 retrieved passage from the off-the-shelf Contriever, while "Updated" means the one from RPDR.
  • Figure 4: The Relationship between augmented data scale and retriever model performance. In the left figure, EQ represents EntityQuestions. The x-axis indicates the proportion of the dataset size relative to the full augmentated dataset.