Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answering

Pascal Tilli, Ngoc Thang Vu

TL;DR

This work introduces an interpretable approach for graph-based VQA and demonstrates competitive performance on the GQA dataset, and presents quantitative metrics that correlate with the evaluations of human assessors, acting as automatic metrics for the generated explanatory subgraphs.

Abstract

The large success of deep learning based methods in Visual Question Answering (VQA) has concurrently increased the demand for explainable methods. Most methods in Explainable Artificial Intelligence (XAI) focus on generating post-hoc explanations rather than taking an intrinsic approach, the latter characterizing an interpretable model. In this work, we introduce an interpretable approach for graph-based VQA and demonstrate competitive performance on the GQA dataset. This approach bridges the gap between interpretability and performance. Our model is designed to intrinsically produce a subgraph during the question-answering process as its explanation, providing insight into the decision making. To evaluate the quality of these generated subgraphs, we compare them against established post-hoc explainability methods for graph neural networks, and perform a human evaluation. Moreover, we present quantitative metrics that correlate with the evaluations of human assessors, acting as automatic metrics for the generated explanatory subgraphs. Our implementation is available at https://github.com/DigitalPhonetics/Intrinsic-Subgraph-Generation-for-VQA.

Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answering

TL;DR

This work introduces an interpretable approach for graph-based VQA and demonstrates competitive performance on the GQA dataset, and presents quantitative metrics that correlate with the evaluations of human assessors, acting as automatic metrics for the generated explanatory subgraphs.

Abstract

The large success of deep learning based methods in Visual Question Answering (VQA) has concurrently increased the demand for explainable methods. Most methods in Explainable Artificial Intelligence (XAI) focus on generating post-hoc explanations rather than taking an intrinsic approach, the latter characterizing an interpretable model. In this work, we introduce an interpretable approach for graph-based VQA and demonstrate competitive performance on the GQA dataset. This approach bridges the gap between interpretability and performance. Our model is designed to intrinsically produce a subgraph during the question-answering process as its explanation, providing insight into the decision making. To evaluate the quality of these generated subgraphs, we compare them against established post-hoc explainability methods for graph neural networks, and perform a human evaluation. Moreover, we present quantitative metrics that correlate with the evaluations of human assessors, acting as automatic metrics for the generated explanatory subgraphs. Our implementation is available at https://github.com/DigitalPhonetics/Intrinsic-Subgraph-Generation-for-VQA.
Paper Structure (45 sections, 3 equations, 8 figures, 7 tables, 1 algorithm)

This paper contains 45 sections, 3 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Neural Networks
  4. Visual Question Answering
  5. Graph based VQA Models
  6. Explainability and Interpretability
  7. Soft- and Hard-Attention
  8. Problem Formulation
  9. Proposed Model Architecture
  10. Question Processing
  11. Scene Graph Encoding
  12. Masking Graph Attention Network
  13. Global Attention Aggregation & Question Answering
  14. Evaluation Methods
  15. Baselines for Comparison
  16. ...and 30 more sections

Figures (8)

  • Figure 1: High-level architecture of our model. The hard attention mask gets computed in -layer four. Only the $top-k$ node embeddings within the hard attention mask are passed in the node attention pooling module.
  • Figure 2: Age and gender distribution of our participants in the user study.
  • Figure 3: Results of the self-assessment regarding the general topics of artificial intelligence (AI) and explainable artificial intelligence (XAI).
  • Figure 4: Graph and image for question Id: 17745707. Question: Is the woman to the left or to the right of the phone?Prediction: left. Ground-truth answer: left. Semantic type: relation. Structural type: choose.
  • Figure 5: Graph and image for question Id: 17267496. Question: Are his eyes large and green? Prediction: no. Ground-truth answer: no. Semantic type: attribute. Structural type: logical.
  • ...and 3 more figures