Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocVXQA: Context-Aware Visual Explanations for Document Question Answering

Mohamed Ali Souibgui, Changkyu Choi, Andrey Barsky, Kangsoo Jung, Ernest Valveny, Dimosthenis Karatzas

TL;DR

DocVXQA tackles the opacity of DocVQA models by learning visually grounded, context-aware explanations via a learnable mask integrated with a DocVQA backbone. The approach leverages an information bottleneck framework to ensure explanations are minimal yet sufficient, augmented by a ColPali prior to guide mask learning and a postprocessing step to produce concise bounding boxes. Key contributions include the first self-explainable DocVQA model, a principled IB-based objective, model-agnostic integration with pretrained architectures, and thorough evaluation including human judgments showing improved interpretability and trustworthy reasoning. The method holds practical significance for transparent document understanding in high-stakes applications, enabling users to inspect and trust model decisions while maintaining competitive QA accuracy.

Abstract

We propose DocVXQA, a novel framework for visually self-explainable document question answering. The framework is designed not only to produce accurate answers to questions but also to learn visual heatmaps that highlight contextually critical regions, thereby offering interpretable justifications for the model's decisions. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning objectives. Unlike conventional methods that emphasize only the regions pertinent to the answer, our framework delivers explanations that are \textit{contextually sufficient} while remaining \textit{representation-efficient}. This fosters user trust while achieving a balance between predictive performance and interpretability in DocVQA applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method. The code is available at https://github.com/dali92002/DocVXQA.

DocVXQA: Context-Aware Visual Explanations for Document Question Answering

TL;DR

DocVXQA tackles the opacity of DocVQA models by learning visually grounded, context-aware explanations via a learnable mask integrated with a DocVQA backbone. The approach leverages an information bottleneck framework to ensure explanations are minimal yet sufficient, augmented by a ColPali prior to guide mask learning and a postprocessing step to produce concise bounding boxes. Key contributions include the first self-explainable DocVQA model, a principled IB-based objective, model-agnostic integration with pretrained architectures, and thorough evaluation including human judgments showing improved interpretability and trustworthy reasoning. The method holds practical significance for transparent document understanding in high-stakes applications, enabling users to inspect and trust model decisions while maintaining competitive QA accuracy.

Abstract

We propose DocVXQA, a novel framework for visually self-explainable document question answering. The framework is designed not only to produce accurate answers to questions but also to learn visual heatmaps that highlight contextually critical regions, thereby offering interpretable justifications for the model's decisions. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning objectives. Unlike conventional methods that emphasize only the regions pertinent to the answer, our framework delivers explanations that are \textit{contextually sufficient} while remaining \textit{representation-efficient}. This fosters user trust while achieving a balance between predictive performance and interpretability in DocVQA applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method. The code is available at https://github.com/dali92002/DocVXQA.
Paper Structure (39 sections, 3 equations, 17 figures, 5 tables)

This paper contains 39 sections, 3 equations, 17 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Document visual question answering
  4. Explainability in vision and language
  5. Learning to explain
  6. Information theoretic learning
  7. Method
  8. Visually self-explainable DocVQA: Enhancing context awareness through a learnable mask
  9. Minimality-sufficiency trade-off
  10. Learning sufficient explanation via cross-entropy
  11. Learning minimal explanation via continuity loss with L1 norm
  12. Learning context-aware explanations through token interactions
  13. Proposed learning objective
  14. Postprocessing
  15. Experiments and Results
  16. ...and 24 more sections

Figures (17)

  • Figure 1: An illustration of the relevant regions in a DocVQA model (highlighted zones), produced by extracting the raw attention maps from the last layer (top) and by using our method (bottom) for the question " 'Pleasure to burn since 1913', Which cigarette's tagline is this?". Here, the answer given correctly by the model is "Camel".
  • Figure 2: Overview of the proposed DocVXQA framework:① The input question and the full document image are sent to both the pretrained Pix2Struct ENC-DEC and a pretrained ColPali model. ② The mask head (MASK) generates a learnable mask based on the decoder output and positional embeddings. ③ ColPali provides a mask prior, highlighting the relevant regions of the input document in relation to the question. ④ The learnable mask, guided by the mask prior, is combined with the original document to create the masked image, where only the highlighted parts are kept visible. ⑤ The masked image is processed through the Pix2Struct network. ⑥ The text head (TEXT) predicts the answer to the question based on the masked input.
  • Figure 3: A comparison of explanations generated by different methods for the question "What is the total amount?" with the model’s answer being "$180,000". Relevance maps with different (thresholds) are applied to the input image to keep only the relevant regions. Best viewed at high zoom. Additional qualitative results across diverse contextual scenarios are provided in \ref{['sec:examples']}.
  • Figure 4: Performance of our method with and without postprocessing, under different thresholds applied to the relevance masks.
  • Figure 5: Visualization of the background removal step in our methods. Question: "What is the estimated budget of ‘conduct analysis of decision makers/ information targets’ in research and development?". Model answer "$ 7,000.00".
  • ...and 12 more figures