Docs2Synth: A Synthetic Data Trained Retriever Framework for Scanned Visually Rich Documents Understanding
Yihao Ding, Qiang Sun, Puzhen Wu, Sirui Li, Siwen Luo, Wei Liu
TL;DR
Docs2Synth addresses the challenge of private-domain VRDU by combining synthetic supervision with retrieval-guided inference. It trains a domain-adapted, lightweight visual retriever on synthetic QA data generated from document collections, then couples this retriever with an MLLM in an iterative retrieval–generation loop to improve grounding and reduce hallucination, formalized over $D$, $T$, $B$, and $L$ through successive iterations $t$ with retrieved content $C^t$ and top-$k$ selections. The framework comprises a Document Processor, Synthetic QA Generator, Retriever Tuning, and Iterative Inference, packaged as an open-source Python toolkit configurable by a single config file. Evaluations on Form-NLU, CORD, and Ephoie show consistent gains in grounding and domain generalization, with the synthetic-data–tuned retriever outperforming strong MLLMs in many settings and qualitative cases illustrating mutual improvements between retriever and LLM components. The work provides a practical pathway for private-domain VRDU by enabling scalable, annotation-free adaptation and easy deployment in real-world systems.
Abstract
Document understanding (VRDU) in regulated domains is particularly challenging, since scanned documents often contain sensitive, evolving, and domain specific knowledge. This leads to two major challenges: the lack of manual annotations for model adaptation and the difficulty for pretrained models to stay up-to-date with domain-specific facts. While Multimodal Large Language Models (MLLMs) show strong zero-shot abilities, they still suffer from hallucination and limited domain grounding. In contrast, discriminative Vision-Language Pre-trained Models (VLPMs) provide reliable grounding but require costly annotations to cover new domains. We introduce Docs2Synth, a synthetic-supervision framework that enables retrieval-guided inference for private and low-resource domains. Docs2Synth automatically processes raw document collections, generates and verifies diverse QA pairs via an agent-based system, and trains a lightweight visual retriever to extract domain-relevant evidence. During inference, the retriever collaborates with an MLLM through an iterative retrieval--generation loop, reducing hallucination and improving response consistency. We further deliver Docs2Synth as an easy-to-use Python package, enabling plug-and-play deployment across diverse real-world scenarios. Experiments on multiple VRDU benchmarks show that Docs2Synth substantially enhances grounding and domain generalization without requiring human annotations.