Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Biomedical Question Answering via Multi-Level Summarization on a Local Knowledge Graph

Lingxiao Guan, Yuanhao Huang, Jie Liu

TL;DR

This work proposes a novel method that utilizes propositional claims to construct a local knowledge graph from retrieved documents and derived via layerwise summarization from the knowledge graph to contextualize a small language model to perform QA.

Abstract

In Question Answering (QA), Retrieval Augmented Generation (RAG) has revolutionized performance in various domains. However, how to effectively capture multi-document relationships, particularly critical for biomedical tasks, remains an open question. In this work, we propose a novel method that utilizes propositional claims to construct a local knowledge graph from retrieved documents. Summaries are then derived via layerwise summarization from the knowledge graph to contextualize a small language model to perform QA. We achieved comparable or superior performance with our method over RAG baselines on several biomedical QA benchmarks. We also evaluated each individual step of our methodology over a targeted set of metrics, demonstrating its effectiveness.

Biomedical Question Answering via Multi-Level Summarization on a Local Knowledge Graph

TL;DR

This work proposes a novel method that utilizes propositional claims to construct a local knowledge graph from retrieved documents and derived via layerwise summarization from the knowledge graph to contextualize a small language model to perform QA.

Abstract

In Question Answering (QA), Retrieval Augmented Generation (RAG) has revolutionized performance in various domains. However, how to effectively capture multi-document relationships, particularly critical for biomedical tasks, remains an open question. In this work, we propose a novel method that utilizes propositional claims to construct a local knowledge graph from retrieved documents. Summaries are then derived via layerwise summarization from the knowledge graph to contextualize a small language model to perform QA. We achieved comparable or superior performance with our method over RAG baselines on several biomedical QA benchmarks. We also evaluated each individual step of our methodology over a targeted set of metrics, demonstrating its effectiveness.

Paper Structure

This paper contains 20 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Retrieval Augmented Generation
  4. Summarization
  5. RAG with Knowledge Graphs
  6. Methods
  7. Relation Extraction
  8. Graph Creation
  9. Graph Summarization
  10. Experiment Settings
  11. Datasets
  12. RAG Datasets
  13. Evaluation Datasets
  14. Model Settings
  15. QA Baselines
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of the proposed layerwise summarization method. Relation extraction: Load in documents with a retriever, break documents into claims, break claims into triples. Graph creation: Build local graph with triples and denoise. Graph summarization: Summarize the graph layerwise with the top re-ranked claims as the roots. The final summaries are provided to a model as context for downstream QA tasks.
  • Figure 2: Layerwise summarization method overview. For a given claim of interest, the graph is organized into layers based on the distance of each connected claim from the claim of interest. The summarization process begins from the furthest layer, moving inwards. For each layer claims are summarized using the previously generated summaries of their connected claims in lower layers. This process ensures that path information and multi-document relationships are preserved while filtering out irrelevant information in the final summaries.

Theorems & Definitions (1)

  • Definition 1: Layer