Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language

Muhammad Asif Ali, Nawal Daftardar, Mutayyaba Waheed, Jianbin Qin, Di Wang

TL;DR

This paper proposes Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL), a new benchmark for rigorous performance evaluation of MQA under KE for Arabic language, and reveals MQA-KEAL outperforms the baseline models by a significant margin.

Abstract

Large Language Models (LLMs) have demonstrated significant capabilities across numerous application domains. A key challenge is to keep these models updated with latest available information, which limits the true potential of these models for the end-applications. Although, there have been numerous attempts for LLMs Knowledge Editing (KE), i.e., to edit the LLMs prior knowledge and in turn test it via Multi-hop Question Answering (MQA), yet so far these studies are primarily focused on English language. To bridge this gap, in this paper we propose: Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL). MQA-KEAL stores knowledge edits as structured knowledge units in the external memory. In order to solve multi-hop question, it first uses task-decomposition to decompose the question into smaller sub-problems. Later for each sub-problem, it iteratively queries the external memory and/or target LLM in order to generate the final response. In addition, we also contribute MQUAKE-AR (Arabic translation of English benchmark MQUAKE), as well as a new benchmark MQA-AEVAL for rigorous performance evaluation of MQA under KE for Arabic language. Experimentation evaluation reveals MQA-KEAL outperforms the baseline models by a significant margin.

MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language

TL;DR

This paper proposes Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL), a new benchmark for rigorous performance evaluation of MQA under KE for Arabic language, and reveals MQA-KEAL outperforms the baseline models by a significant margin.

Abstract

Large Language Models (LLMs) have demonstrated significant capabilities across numerous application domains. A key challenge is to keep these models updated with latest available information, which limits the true potential of these models for the end-applications. Although, there have been numerous attempts for LLMs Knowledge Editing (KE), i.e., to edit the LLMs prior knowledge and in turn test it via Multi-hop Question Answering (MQA), yet so far these studies are primarily focused on English language. To bridge this gap, in this paper we propose: Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL). MQA-KEAL stores knowledge edits as structured knowledge units in the external memory. In order to solve multi-hop question, it first uses task-decomposition to decompose the question into smaller sub-problems. Later for each sub-problem, it iteratively queries the external memory and/or target LLM in order to generate the final response. In addition, we also contribute MQUAKE-AR (Arabic translation of English benchmark MQUAKE), as well as a new benchmark MQA-AEVAL for rigorous performance evaluation of MQA under KE for Arabic language. Experimentation evaluation reveals MQA-KEAL outperforms the baseline models by a significant margin.
Paper Structure (45 sections, 6 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 45 sections, 6 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Notation
  5. Problem Definition
  6. MQA-Keal
  7. Structured knowledge Retrieval
  8. Task Decomposition
  9. Iterative Traversal over $\mathcal{P}$
  10. Response from Structured Retrieval
  11. Response via target LLM
  12. Experimentation
  13. Experimental Settings
  14. Datasets.
  15. Experimental Setup.
  16. ...and 30 more sections

Figures (6)

  • Figure 1: An example illustration of multi-hop question answering under knowledge editing for Arabic and English language.
  • Figure 2: Workflow of MQA-Keal. The left part of the Figure shows how we store fact edits. The central part illustrates task decomposition and candidate generation from the memory. The right part of the Figure shows candidate refinement and final response generation.
  • Figure 3: An example illustrating the limitation of existing memory-based methods that store knowledge edits as unstructured text, vs structured knowledge retrieval employed by MQA-Keal.
  • Figure 4: An example illustration of MQA-Aeval.
  • Figure :
  • ...and 1 more figures