Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Review-Then-Refine: A Dynamic Framework for Multi-Hop Question Answering with Temporal Adaptability

Xiangsen Chen, Xuming Hu, Nan Tang

TL;DR

The paper addresses the challenge of multi-hop question answering with temporal information by introducing a dynamic two-phase framework, review-then-refine. The review phase dynamically rewrites and decomposes complex queries, guided by a retrieval gate that minimizes unnecessary external fetches, while the refine phase synthesizes retrieved data with the model's internal knowledge to produce coherent, time-sensitive answers. Across dynamic and static QA benchmarks, the approach achieves state-of-the-art performance on several datasets and demonstrates the critical role of dynamic rewriting, query decomposition, and adaptive retrieval. The work advances practical QA systems by reducing hallucinations and improving temporal consistency, with potential impact on real-time information synthesis and knowledge-intensive tasks. Limitations include dependence on external source quality and runtime efficiency, suggesting future work on more scalable retrieval strategies and domain-specific adaptations.

Abstract

Retrieve-augmented generation (RAG) frameworks have emerged as a promising solution to multi-hop question answering(QA) tasks since it enables large language models (LLMs) to incorporate external knowledge and mitigate their inherent knowledge deficiencies. Despite this progress, existing RAG frameworks, which usually follows the retrieve-then-read paradigm, often struggle with multi-hop QA with temporal information since it has difficulty retrieving and synthesizing accurate time-related information. To address the challenge, this paper proposes a novel framework called review-then-refine, which aims to enhance LLM performance in multi-hop QA scenarios with temporal information. Our approach begins with a review phase, where decomposed sub-queries are dynamically rewritten with temporal information, allowing for subsequent adaptive retrieval and reasoning process. In addition, we implement adaptive retrieval mechanism to minimize unnecessary retrievals, thus reducing the potential for hallucinations. In the subsequent refine phase, the LLM synthesizes the retrieved information from each sub-query along with its internal knowledge to formulate a coherent answer. Extensive experimental results across multiple datasets demonstrate the effectiveness of our proposed framework, highlighting its potential to significantly improve multi-hop QA capabilities in LLMs.

Review-Then-Refine: A Dynamic Framework for Multi-Hop Question Answering with Temporal Adaptability

TL;DR

The paper addresses the challenge of multi-hop question answering with temporal information by introducing a dynamic two-phase framework, review-then-refine. The review phase dynamically rewrites and decomposes complex queries, guided by a retrieval gate that minimizes unnecessary external fetches, while the refine phase synthesizes retrieved data with the model's internal knowledge to produce coherent, time-sensitive answers. Across dynamic and static QA benchmarks, the approach achieves state-of-the-art performance on several datasets and demonstrates the critical role of dynamic rewriting, query decomposition, and adaptive retrieval. The work advances practical QA systems by reducing hallucinations and improving temporal consistency, with potential impact on real-time information synthesis and knowledge-intensive tasks. Limitations include dependence on external source quality and runtime efficiency, suggesting future work on more scalable retrieval strategies and domain-specific adaptations.

Abstract

Retrieve-augmented generation (RAG) frameworks have emerged as a promising solution to multi-hop question answering(QA) tasks since it enables large language models (LLMs) to incorporate external knowledge and mitigate their inherent knowledge deficiencies. Despite this progress, existing RAG frameworks, which usually follows the retrieve-then-read paradigm, often struggle with multi-hop QA with temporal information since it has difficulty retrieving and synthesizing accurate time-related information. To address the challenge, this paper proposes a novel framework called review-then-refine, which aims to enhance LLM performance in multi-hop QA scenarios with temporal information. Our approach begins with a review phase, where decomposed sub-queries are dynamically rewritten with temporal information, allowing for subsequent adaptive retrieval and reasoning process. In addition, we implement adaptive retrieval mechanism to minimize unnecessary retrievals, thus reducing the potential for hallucinations. In the subsequent refine phase, the LLM synthesizes the retrieved information from each sub-query along with its internal knowledge to formulate a coherent answer. Extensive experimental results across multiple datasets demonstrate the effectiveness of our proposed framework, highlighting its potential to significantly improve multi-hop QA capabilities in LLMs.

Paper Structure

This paper contains 21 sections, 6 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Hop Question Answering
  4. Retrieval-Augmented Generation for LLMs
  5. Review-then-Refine
  6. Review
  7. Refine
  8. Experiments and Analysis
  9. Datasets
  10. Baselines
  11. Evaluation
  12. Results
  13. Ablation Study
  14. Conclusion
  15. Implementation Detail
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Illustration of challenges in multi-hop QA with temporal information and how our proposed review-then-refine method addresses them. The traditional retrieval-then-read paradigm often introduces external hallucinations or retrieves wrong information, while the review-then-refine method dynamically adjusts retrieval and query rewriting to produce the correct answer.
  • Figure 2: The illustration of how the framework works.