Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Long-Context Language Models Better Multi-Hop Reasoners

Yanyang Li, Shuo Liang, Michael R. Lyu, Liwei Wang

TL;DR

Long-context language models struggle with multi-hop reasoning in noisy contexts. The paper proposes Reasoning with Attributions, prompting models to ground each assertion to source-context via Chain-of-Citation or Chain-of-Quote, and provides MuSiQue-Attribute plus a mixed multi-task learning and data augmentation strategy to train attribution-aware reasoning. Empirical results across three multi-hop benchmarks show attribution-based prompting improves performance and robustness, with a Vicuna-7B variant trained with AttrLoRA achieving competitive results with ChatGPT and Claude-instant on MuSiQue. This work offers a viable path to more reliable long-context reasoning and releases resources to foster further research.

Abstract

Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.

Making Long-Context Language Models Better Multi-Hop Reasoners

TL;DR

Long-context language models struggle with multi-hop reasoning in noisy contexts. The paper proposes Reasoning with Attributions, prompting models to ground each assertion to source-context via Chain-of-Citation or Chain-of-Quote, and provides MuSiQue-Attribute plus a mixed multi-task learning and data augmentation strategy to train attribution-aware reasoning. Empirical results across three multi-hop benchmarks show attribution-based prompting improves performance and robustness, with a Vicuna-7B variant trained with AttrLoRA achieving competitive results with ChatGPT and Claude-instant on MuSiQue. This work offers a viable path to more reliable long-context reasoning and releases resources to foster further research.

Abstract

Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.
Paper Structure (23 sections, 6 figures, 11 tables)

This paper contains 23 sections, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Reasoning with Attributions
  3. Pilot Study
  4. Dataset Curation
  5. Learning to Attribute in Reasoning
  6. Multi-Task Learning.
  7. Data Augmentation.
  8. Experiments
  9. Datasets
  10. Models
  11. Main Results
  12. Analysis
  13. Related Work
  14. Multi-Hop Reasoning.
  15. Context Utilization.
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Comparison of the proposed auxiliary tasks.
  • Figure 2: Exact-Match (EM) results of different models under various noise levels in three multi-hop reasoning datasets. Note that all models except our AttrLoRA use 5-shot prompting. A higher noise ratio indicates more distractors, i.e., irrelevant documents, are presented in the context of both the test instance and the demonstrations.
  • Figure 3: Multi-hop reasoning performance vs. citation precision and recall of AttrLoRA.
  • Figure 4: The impact of scaling fine-tuning data size.
  • Figure 5: Screenshot of our human annotation tool.
  • ...and 1 more figures