WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?

TL;DR

WildDoc introduces the first real-world document understanding benchmark, assembling over 12k manually captured document images across diverse environments, illumination, views, distortions, and effects. By leveraging sources from DocVQA, ChartQA, and TableVQA, it enables fair cross-benchmark comparisons and introduces a Consistency Score to quantify robustness across four capture conditions. Evaluations of leading MLLMs reveal substantial performance drops and limited robustness in real-world conditions, with notable dispersion among models that perform similarly on digital benchmarks. The work also analyzes distortion-related degradation and proposes practical strategies—data augmentation, robust feature representations, preprocessing rectification, and expanded real-world data—to advance in-the-wild document understanding.

Abstract

The rapid advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced capabilities in Document Understanding. However, prevailing benchmarks like DocVQA and ChartQA predominantly comprise \textit{scanned or digital} documents, inadequately reflecting the intricate challenges posed by diverse real-world scenarios, such as variable illumination and physical distortions. This paper introduces WildDoc, the inaugural benchmark designed specifically for assessing document understanding in natural environments. WildDoc incorporates a diverse set of manually captured document images reflecting real-world conditions and leverages document sources from established benchmarks to facilitate comprehensive comparisons with digital or scanned documents. Further, to rigorously evaluate model robustness, each document is captured four times under different conditions. Evaluations of state-of-the-art MLLMs on WildDoc expose substantial performance declines and underscore the models' inadequate robustness compared to traditional benchmarks, highlighting the unique challenges posed by real-world document understanding. Our project homepage is available at https://bytedance.github.io/WildDoc.

Figures (7)

• Figure 1: Comparison of WildDoc with existing benchmarks for document understanding, highlighting the predominance of scanned or digital document images in current benchmarks versus the real-world captured document images in WildDoc.
• Figure 2: Overview of the WildDoc. (a) For every document, we manually capture four images under different setups. (b) Several representative examples that encompass different real-world conditions. More examples are listed in the Appendix.
• Figure 3: Statistics on image capture setup.
• Figure 4: Statistics on the image capture equipment.
• Figure 5: Evaluation results of Qwen2.5-VL-72B in the Original DocVQA mathew2021docvqa and our WildDoc benchmark. The answer in the figure is highlighted in red. Zoom in for the best view.