Where is this coming from? Making groundedness count in the evaluation of Document VQA models

TL;DR

The paper identifies a gap in DocVQA evaluation: standard metrics like ANLS ignore whether answers are semantically typed and explicitly grounded in the input document. It introduces SMuDGE, a configurable composite metric combining a multimodal grounding score $g_i$ and a type-aware surface similarity score $m_i$ via $s_i = \alpha m_i + (1-\alpha) g_i$, with $g_i$ computed from bounding-box localization and $m_i$ guided by output type, including a numeric-tolerant, textual, and hybrid handling. Across DocVQA, InfographicVQA, MP-DocVQA, and DUDE, SMuDGE better aligns with human judgments and model calibration, producing meaningful leaderboard re-rankings that emphasize grounded and well-calibrated outputs; an optimal balance near $\alpha \approx 0.25$ is reported. The results suggest that grounding-aware evaluation can promote more robust, trustworthy document understanding models and guide improvements beyond surface similarity alone.

Abstract

Document Visual Question Answering (VQA) models have evolved at an impressive rate over the past few years, coming close to or matching human performance on some benchmarks. We argue that common evaluation metrics used by popular benchmarks do not account for the semantic and multimodal groundedness of a model's outputs. As a result, hallucinations and major semantic errors are treated the same way as well-grounded outputs, and the evaluation scores do not reflect the reasoning capabilities of the model. In response, we propose a new evaluation methodology that accounts for the groundedness of predictions with regard to the semantic characteristics of the output as well as the multimodal placement of the output within the input document. Our proposed methodology is parameterized in such a way that users can configure the score according to their preferences. We validate our scoring methodology using human judgment and show its potential impact on existing popular leaderboards. Through extensive analyses, we demonstrate that our proposed method produces scores that are a better indicator of a model's robustness and tends to give higher rewards to better-calibrated answers.

Figures (15)

• Figure 1: Two excerpts from an image document from the DocVQA dataset docvqa2021.
• Figure 2: The rankings of the top 10 models on the DocVQA leaderboard, before and after applying our composite score with $\alpha = 0.25$. Left segment: Rankings based on ANLS versus our score. Middle segment: Our rankings broken down by question type. Right segment: Our rankings broken down by answer type.
• Figure 3: The correlation between the rankings produced by our method (with $\alpha=0.25$) and the original ANLS-based ranking, broken down by the type of answer. All $\tau$ values are significant at $p \ll 0.05$.
• Figure 4: Kendall's $\tau$ rank correlation with the original DocVQA leaderboard, broken down by question types. All $\tau$ values are significant at $p \ll 0.05$.
• Figure 5: Pearson $R$ correlation with the calibration error of models based on the DUDE leaderboard, broken down by answer type.