Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee

TL;DR

This work proposes EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs, and shows that EFSum improves LLM’s zero-shot QA performance with its helpful and faithful summaries, especially when noisy facts are retrieved.

Abstract

Recent studies have investigated utilizing Knowledge Graphs (KGs) to enhance Quesetion Answering (QA) performance of Large Language Models (LLMs), yet structured KG verbalization remains challengin. Existing methods, such as triple-form or free-form textual conversion of triple-form facts, encounter several issues. These include reduced evidence density due to duplicated entities or relationships, and reduced evidence clarity due to an inability to emphasize crucial evidence. To address these issues, we propose EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs. We optimize an open-source LLM as a fact summarizer through distillation and preference alignment. Our extensive experiments show that EFSum improves LLM's zero-shot QA performance, and it is possible to ensure both the helpfulness and faithfulness of the summary.

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

TL;DR

This work proposes EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs, and shows that EFSum improves LLM’s zero-shot QA performance with its helpful and faithful summaries, especially when noisy facts are retrieved.

Abstract

Recent studies have investigated utilizing Knowledge Graphs (KGs) to enhance Quesetion Answering (QA) performance of Large Language Models (LLMs), yet structured KG verbalization remains challengin. Existing methods, such as triple-form or free-form textual conversion of triple-form facts, encounter several issues. These include reduced evidence density due to duplicated entities or relationships, and reduced evidence clarity due to an inability to emphasize crucial evidence. To address these issues, we propose EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs. We optimize an open-source LLM as a fact summarizer through distillation and preference alignment. Our extensive experiments show that EFSum improves LLM's zero-shot QA performance, and it is possible to ensure both the helpfulness and faithfulness of the summary.
Paper Structure (36 sections, 2 equations, 5 figures, 10 tables)

This paper contains 36 sections, 2 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. KG-Augmented LLM Prompting for QA
  4. Analysis on Verbalized Facts
  5. EFSum: Proposed Method
  6. LLM Prompting for EFSum
  7. LLM Fine-Tuning for EFSum
  8. Distillation of Fact Summarization
  9. Alignment with Summary Preference
  10. Experiments
  11. Experimental Settings
  12. Effectiveness of Dense Evidence (RQ1)
  13. Effectiveness of Clear Evidence (RQ2)
  14. Effect of Preference Alignment (RQ3)
  15. Related Work
  16. ...and 21 more sections

Figures (5)

  • Figure 1: The QA pipeline based on LLM prompting, augmented with relevant facts from KGs. Our fact summarization improves both density and clarity of evidence within contextual knowledge for enhanced QA.
  • Figure 2: Analysis on each fact verbalization method.
  • Figure 3: The overall framework of EFSum$distill$. Our fact summarizer is trained to generate evidence-focused summaries via LLM distillation, and then further optimized to align the QA-specific preference, which enhances the helpfulness and faithfulness of its output summaries.
  • Figure 4: Summary-level and answer-level QA accuracies with respect to the number of relevant facts on WebQSP (Upper) and Mintaka (Lower), respectively.
  • Figure 5: Two quality metrics of verbalized facts.