Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding

TL;DR

This work tackles visual document understanding by addressing the lack of contextual grounding in multimodal models. It introduces an adaptive markup language generation framework that translates documents into structured markup formats (plain text, Markdown, LaTeX, HTML, JSON, TikZ) to provide explicit context for reasoning, backed by DocMark-Pile (3.8M pretraining pairs) and DocMark-Instruct (624k CoT-style annotations). The approach uses a two-round, chain-of-thought-inspired pipeline where the model first outputs a markup-based context and then the final answer, enabling better disambiguation of complex layouts and content. Empirical results show substantial improvements over state-of-the-art MLLMs across a range of markup-generation and downstream document-understanding tasks, with detailed ablations confirming the value of the pretraining and CoT fine-tuning components; code and models are released for broader adoption.

Abstract

Visual Document Understanding has become essential with the increase of text-rich visual content. This field poses significant challenges due to the need for effective integration of visual perception and textual comprehension, particularly across diverse document types with complex layouts. Moreover, existing fine-tuning datasets for this domain often fall short in providing the detailed contextual information for robust understanding, leading to hallucinations and limited comprehension of spatial relationships among visual elements. To address these challenges, we propose an innovative pipeline that utilizes adaptive generation of markup languages, such as Markdown, JSON, HTML, and TiKZ, to build highly structured document representations and deliver contextually-grounded responses. We introduce two fine-grained structured datasets: DocMark-Pile, comprising approximately 3.8M pretraining data pairs for document parsing, and DocMark-Instruct, featuring 624k fine-tuning data annotations for grounded instruction following. Extensive experiments demonstrate that our proposed model significantly outperforms existing state-of-theart MLLMs across a range of visual document understanding benchmarks, facilitating advanced reasoning and comprehension capabilities in complex visual scenarios. Our code and models are released at https://github. com/Euphoria16/DocMark.

Figures (10)

• Figure 1: Comparison of existing Data annotations and model Predictions: InternVL2-2B vs. our DocMark-2B. Existing datasets consist primarily of questions and brief answers, often resulting in model hallucinations. In contrast, our model effectively converts documents into structured markup languages, offering critical contextual information that enhances the accurate interpretation of the content.
• Figure 2: Overview of our DocMark-Pile dataset for pretraining on various markup language generation tasks. Due to the length of the answer, we have omitted a part of the content for better display.
• Figure 3: Domain distribution of our DocMark-Pile dataset. (a) Distributions of different markup language parsing tasks. (b) Distributions of different image types.
• Figure 4: Overview of our adaptive markup language generation framework. (a) During pretraining, the model learns to parse documents into corresponding markup language representations, enhancing its structural understanding capabilities. (b) In the finetuning phase, the model is optimized to identify and extract relevant information as intermediate rationales to formulate precise answers.
• Figure 5: Ablation study on employing different training dataset strategies.