Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions

Naoki Shimoda, Akihiro Yamamoto

TL;DR

This work tackles the reliability of MCQA by fusing Transformer-based relation extraction with knowledge-graph matching to verify entailment in a traceable, graph-based framework. By transforming each question option into a propositional graph and grounding its assertions in a Wikipedia-derived KG, the method computes a truth score that guides answer selection while revealing the verification path. Experiments on two 4-way cloze MCQ datasets show the approach can reach up to about 70% accuracy, with performance strongly influenced by question category and the efficacy of entity linking. The work advances explainable AI in MCQA by making the reasoning process auditable through explicit graph structures and verified edges, suggesting paths for improving accuracy via better article retrieval and more flexible reasoning.

Abstract

In this research, we combine Transformer-based relation extraction with matching of knowledge graphs (KGs) and apply them to answering multiple-choice questions (MCQs) while maintaining the traceability of the output process. KGs are structured representations of factual knowledge consisting of entities and relations. Due to the high construction cost, they had been regarded as static databases with validated links. However, the recent development of Transformer-based relation extraction (RE) methods has enabled us to generate KGs dynamically by giving them natural language texts, and thereby opened the possibility for representing the meaning of the input sentences with the created KGs. Using this effect, we propose a method that answers MCQs in the "fill-in-the-blank" format, taking care of the point that RE methods generate KGs that represent false information if provided with factually incorrect texts. We measure the truthfulness of each question sentence by (i) converting the sentence into a relational graph using an RE method and (ii) verifying it against factually correct KGs under the closed-world assumption. The experimental results demonstrate that our method correctly answers up to around 70% of the questions, while providing traceability of the procedure. We also highlight that the question category has a vast influence on the accuracy.

Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions

TL;DR

This work tackles the reliability of MCQA by fusing Transformer-based relation extraction with knowledge-graph matching to verify entailment in a traceable, graph-based framework. By transforming each question option into a propositional graph and grounding its assertions in a Wikipedia-derived KG, the method computes a truth score that guides answer selection while revealing the verification path. Experiments on two 4-way cloze MCQ datasets show the approach can reach up to about 70% accuracy, with performance strongly influenced by question category and the efficacy of entity linking. The work advances explainable AI in MCQA by making the reasoning process auditable through explicit graph structures and verified edges, suggesting paths for improving accuracy via better article retrieval and more flexible reasoning.

Abstract

In this research, we combine Transformer-based relation extraction with matching of knowledge graphs (KGs) and apply them to answering multiple-choice questions (MCQs) while maintaining the traceability of the output process. KGs are structured representations of factual knowledge consisting of entities and relations. Due to the high construction cost, they had been regarded as static databases with validated links. However, the recent development of Transformer-based relation extraction (RE) methods has enabled us to generate KGs dynamically by giving them natural language texts, and thereby opened the possibility for representing the meaning of the input sentences with the created KGs. Using this effect, we propose a method that answers MCQs in the "fill-in-the-blank" format, taking care of the point that RE methods generate KGs that represent false information if provided with factually incorrect texts. We measure the truthfulness of each question sentence by (i) converting the sentence into a relational graph using an RE method and (ii) verifying it against factually correct KGs under the closed-world assumption. The experimental results demonstrate that our method correctly answers up to around 70% of the questions, while providing traceability of the procedure. We also highlight that the question category has a vast influence on the accuracy.

Paper Structure

This paper contains 21 sections, 15 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Multiple Choice Question Answering
  3. Preliminaries
  4. Methodology
  5. Step 1.
  6. Step 2.
  7. Step 3.
  8. Step 4.
  9. Experiments
  10. Experimental Setup
  11. MCQ Datasets
  12. Relation Extraction Methods
  13. Entity Linking Method
  14. Semantic Similarity Measure
  15. Wikipedia Articles
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Accuracy per category for REBEL with entity linking on KR-200m. The columns with Correct, Incorrect, and Unselectable show the results without random choice, determined solely by the edge and node scores. Stochastic shows the accuracy with random selection for the unselectable questions.
  • Figure 2: PGs for the correct option $o_1\colon$"Starry Night" (left) and the incorrect option $o_4\colon$"The Scream" (right).

Theorems & Definitions (4)

  • Example 1: Barack Obama
  • Definition 1: Cloze test 4-way MCQA
  • Example 2: Relation Extraction
  • Example 3: Starry Night