Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Benchmarking through the Lense of Concept Learning

Quannian Zhang, Michael Röder, Nikit Srivastava, N'Dah Jean Kouagou, Axel-Cyrille Ngonga Ngomo

TL;DR

This work introduces explainable benchmarking, a paradigm that automatically generates human-understandable explanations for benchmark results. It formalizes a knowledge-graph QA setting and presents PruneCEL, a scalable concept-learning algorithm that uses a downward refinement operator with a SPARQL-backed oracle to produce concise, interpretable explanations for when QA systems perform well or poorly. Empirical results show PruneCEL-RS outperforms state-of-the-art concept learners on large benchmarks (e.g., QALD-derived KBs) and a user study with 41 participants indicates explanations help people anticipate system behavior in most cases. The approach promises actionable insights for benchmark design and system improvement, with open-source code and data provided for replication and extension.

Abstract

Evaluating competing systems in a comparable way, i.e., benchmarking them, is an undeniable pillar of the scientific method. However, system performance is often summarized via a small number of metrics. The analysis of the evaluation details and the derivation of insights for further development or use remains a tedious manual task with often biased results. Thus, this paper argues for a new type of benchmarking, which is dubbed explainable benchmarking. The aim of explainable benchmarking approaches is to automatically generate explanations for the performance of systems in a benchmark. We provide a first instantiation of this paradigm for knowledge-graph-based question answering systems. We compute explanations by using a novel concept learning approach developed for large knowledge graphs called PruneCEL. Our evaluation shows that PruneCEL outperforms state-of-the-art concept learners on the task of explainable benchmarking by up to 0.55 points F1 measure. A task-driven user study with 41 participants shows that in 80\% of the cases, the majority of participants can accurately predict the behavior of a system based on our explanations. Our code and data are available at https://github.com/dice-group/PruneCEL/tree/K-cap2025

Explainable Benchmarking through the Lense of Concept Learning

TL;DR

This work introduces explainable benchmarking, a paradigm that automatically generates human-understandable explanations for benchmark results. It formalizes a knowledge-graph QA setting and presents PruneCEL, a scalable concept-learning algorithm that uses a downward refinement operator with a SPARQL-backed oracle to produce concise, interpretable explanations for when QA systems perform well or poorly. Empirical results show PruneCEL-RS outperforms state-of-the-art concept learners on large benchmarks (e.g., QALD-derived KBs) and a user study with 41 participants indicates explanations help people anticipate system behavior in most cases. The approach promises actionable insights for benchmark design and system improvement, with open-source code and data provided for replication and extension.

Abstract

Evaluating competing systems in a comparable way, i.e., benchmarking them, is an undeniable pillar of the scientific method. However, system performance is often summarized via a small number of metrics. The analysis of the evaluation details and the derivation of insights for further development or use remains a tedious manual task with often biased results. Thus, this paper argues for a new type of benchmarking, which is dubbed explainable benchmarking. The aim of explainable benchmarking approaches is to automatically generate explanations for the performance of systems in a benchmark. We provide a first instantiation of this paradigm for knowledge-graph-based question answering systems. We compute explanations by using a novel concept learning approach developed for large knowledge graphs called PruneCEL. Our evaluation shows that PruneCEL outperforms state-of-the-art concept learners on the task of explainable benchmarking by up to 0.55 points F1 measure. A task-driven user study with 41 participants shows that in 80\% of the cases, the majority of participants can accurately predict the behavior of a system based on our explanations. Our code and data are available at https://github.com/dice-group/PruneCEL/tree/K-cap2025
Paper Structure (27 sections, 12 equations, 3 figures, 5 tables)

This paper contains 27 sections, 12 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related-work
  3. QA Benchmarking
  4. Concept Learning
  5. Preliminaries
  6. Description Logics
  7. Refinement Operators
  8. Concept Learning
  9. Knowledge-Graph-based Question Answering
  10. Explainable Benchmarking
  11. Approach
  12. PruneCEL
  13. Refinement Operator
  14. Oracle Implementation
  15. Heuristic
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overview over the three steps of our approach.
  • Figure 2: An excerpt of the knowledge base around the question dqq:Q11 (question 11 from QALD-10). Wikidata entities and properties are replaced by their labels.
  • Figure 3: Examples of a classic (top) and a PruneCEL (bottom) search tree. The green expressions are further refined. Expressions with a dotted frame do not have any given example as instance. The labels on the edges are the generated templates used to derive concepts or roles from the oracle.