Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoTAR: Chain-of-Thought Attribution Reasoning with Multi-level Granularity

Moshe Berchansky, Daniel Fleischer, Moshe Wasserblat, Peter Izsak

TL;DR

This work introduces an attribution-oriented Chain-of-Thought reasoning method to enhance the accuracy of attributions and enhances the response and attribution accuracy of two smaller LLMs, showing their potential to outperform GPT-4 in some cases.

Abstract

State-of-the-art performance in QA tasks is currently achieved by systems employing Large Language Models (LLMs), however these models tend to hallucinate information in their responses. One approach focuses on enhancing the generation process by incorporating attribution from the given input to the output. However, the challenge of identifying appropriate attributions and verifying their accuracy against a source is a complex task that requires significant improvements in assessing such systems. We introduce an attribution-oriented Chain-of-Thought reasoning method to enhance the accuracy of attributions. This approach focuses the reasoning process on generating an attribution-centric output. Evaluations on two context-enhanced question-answering datasets using GPT-4 demonstrate improved accuracy and correctness of attributions. In addition, the combination of our method with finetuning enhances the response and attribution accuracy of two smaller LLMs, showing their potential to outperform GPT-4 in some cases.

CoTAR: Chain-of-Thought Attribution Reasoning with Multi-level Granularity

TL;DR

This work introduces an attribution-oriented Chain-of-Thought reasoning method to enhance the accuracy of attributions and enhances the response and attribution accuracy of two smaller LLMs, showing their potential to outperform GPT-4 in some cases.

Abstract

State-of-the-art performance in QA tasks is currently achieved by systems employing Large Language Models (LLMs), however these models tend to hallucinate information in their responses. One approach focuses on enhancing the generation process by incorporating attribution from the given input to the output. However, the challenge of identifying appropriate attributions and verifying their accuracy against a source is a complex task that requires significant improvements in assessing such systems. We introduce an attribution-oriented Chain-of-Thought reasoning method to enhance the accuracy of attributions. This approach focuses the reasoning process on generating an attribution-centric output. Evaluations on two context-enhanced question-answering datasets using GPT-4 demonstrate improved accuracy and correctness of attributions. In addition, the combination of our method with finetuning enhances the response and attribution accuracy of two smaller LLMs, showing their potential to outperform GPT-4 in some cases.
Paper Structure (18 sections, 1 figure, 9 tables, 1 algorithm)

This paper contains 18 sections, 1 figure, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Attribution-Oriented Question Answering
  4. Chain-of-Thought Attribution Reasoning
  5. Evaluation Metrics
  6. Experimental Setup
  7. Datasets
  8. Span Dataset Preparation
  9. Models
  10. Method Comparison
  11. Experimental Results
  12. Related Work
  13. Conclusions
  14. Limitations
  15. Potential Risks
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Usage of CoT for attribution-based answers. We either instruct the model, using fewshot examples, or finetune the model, to produce a detailed list of the salient information from each passage. Each entry can be either on the passage, sentence, and even span level. Finally, the model produces a coherent and faithful answer.