Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering

Mingda Chen, Xilun Chen, Wen-tau Yih

TL;DR

This work tackles open-domain multi-hop QA under limited labeled data by introducing a data-synthesis framework that leverages naturally occurring document relationships to generate millions of QA-item and retrieval data using prompts on a frozen LLaMA-65B. The synthesized data are used to finetune much smaller models (7B and 65B LLaMA), achieving large performance gains and approaching GPT-3.5-level results while using far fewer parameters. The approach extends to fact verification (FEVER) and is shown to benefit from diverse document relationships and robust query verification, with substantial gains observed even at modest data scales. Overall, the method demonstrates that carefully constructed, multi-function data generation driven by LLM prompts can substantially reduce the need for large models and labeled data in open-domain multi-hop reasoning tasks, enabling practical, scalable systems. Key findings include dramatic improvements for 7B models, competitive performance with GPT-3.5 for larger models, and the critical roles of filtering and relationship diversity in data synthesis.

Abstract

Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models (LLMs). While powerful, these LLMs usually contain tens or hundreds of billions of parameters, making them rather inefficient at inference time. To improve performance of smaller language models, we propose a data synthesis framework for multi-hop question answering that requires less than 10 human annotated question answer pairs. Our framework depends only on rich, naturally-occurring relationships among documents and is built upon the data generation functions parameterized by LLMs and prompts. We synthesize millions of multi-hop questions and claims to finetune language models, evaluated on popular benchmarks for multi-hop question answering and fact verification. Empirically, our approach improves model performance significantly, allowing the finetuned models to be competitive with GPT-3.5 based approaches while being almost one-third the size in parameter count.

Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering

TL;DR

This work tackles open-domain multi-hop QA under limited labeled data by introducing a data-synthesis framework that leverages naturally occurring document relationships to generate millions of QA-item and retrieval data using prompts on a frozen LLaMA-65B. The synthesized data are used to finetune much smaller models (7B and 65B LLaMA), achieving large performance gains and approaching GPT-3.5-level results while using far fewer parameters. The approach extends to fact verification (FEVER) and is shown to benefit from diverse document relationships and robust query verification, with substantial gains observed even at modest data scales. Overall, the method demonstrates that carefully constructed, multi-function data generation driven by LLM prompts can substantially reduce the need for large models and labeled data in open-domain multi-hop reasoning tasks, enabling practical, scalable systems. Key findings include dramatic improvements for 7B models, competitive performance with GPT-3.5 for larger models, and the critical roles of filtering and relationship diversity in data synthesis.

Abstract

Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models (LLMs). While powerful, these LLMs usually contain tens or hundreds of billions of parameters, making them rather inefficient at inference time. To improve performance of smaller language models, we propose a data synthesis framework for multi-hop question answering that requires less than 10 human annotated question answer pairs. Our framework depends only on rich, naturally-occurring relationships among documents and is built upon the data generation functions parameterized by LLMs and prompts. We synthesize millions of multi-hop questions and claims to finetune language models, evaluated on popular benchmarks for multi-hop question answering and fact verification. Empirically, our approach improves model performance significantly, allowing the finetuned models to be competitive with GPT-3.5 based approaches while being almost one-third the size in parameter count.
Paper Structure (33 sections, 5 figures, 16 tables)

This paper contains 33 sections, 5 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Synthesis using Language Models.
  4. Question/Query Generation.
  5. Prompting for Multi-Hop Question Answering.
  6. Knowledge Distillation.
  7. Approach
  8. Data Preparation
  9. Question Generation
  10. Question Answering
  11. Query Generation
  12. Query Verification
  13. Extend to Fact Verification
  14. Experiment
  15. Setup
  16. ...and 18 more sections

Figures (5)

  • Figure 1: An illustration of the overall pipeline of our proposed approach. Each data instance in our synthesized dataset consists of a question, queries and their corresponding retrieved documents, and an answer. We first prompt LLMs to synthesize questions and queries, finetune models on the synthesized data, and then evaluate the finetuned models on downstream tasks that require iteratively querying retrieval corpora.
  • Figure 2: Prompt excerpts for the question generation task for the "hyper" setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in \ref{['appendix-sec:prompts']}.
  • Figure 3: Prompt excerpts for the question answering task for the "hyper" setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in \ref{['appendix-sec:prompts']}.
  • Figure 4: Prompt excerpts for the query generation task for the "hyper" setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in \ref{['appendix-sec:prompts']}.
  • Figure 5: Average dataset performance for HotpotQA, MuSiQue, 2WikiQA, and FEVER. We vary the amount of finetuning data and model sizes. We report model performance using SelfAsk when the amount of finetuning data equals to zero.