DAVE: A VLM Vision Encoder for Document Understanding and Web Agents

TL;DR

DAVE introduces a purpose-built vision encoder for document and web understanding, addressing the gap left by natural-image-focused encoders. It employs a two-stage pretraining: self-supervised MAE on 20M unlabeled document/web images to capture structural cues, followed by supervised autoregressive pretraining with multi-decoders; weight merging creates a decoder-agnostic encoder, while ensemble training fuses domain-specific and general features. Thorough evaluations across classic document tasks, vision-language benchmarks, and web-agent scenarios (including Mind2Web) show consistent gains over strong baselines, averaging ~10.5% on doc/web VLM tasks and ~5% on agentic tasks. The approach demonstrates data-efficient adaptation to specialized domains and provides a scalable pathway for document/web intelligent systems, albeit with fixed resolutions and high compute demands.

Abstract

While Vision-language models (VLMs) have demonstrated remarkable performance across multi-modal tasks, their choice of vision encoders presents a fundamental weakness: their low-level features lack the robust structural and spatial information essential for document understanding and web agents. To bridge this gap, we introduce DAVE, a vision encoder purpose-built for VLMs and tailored for these tasks. Our training pipeline is designed to leverage abundant unlabeled data to bypass the need for costly large-scale annotations for document and web images. We begin with a self-supervised pretraining stage on unlabeled images, followed by a supervised autoregressive pretraining stage, where the model learns tasks like parsing and localization from limited, high-quality data. Within the supervised stage, we adopt two strategies to improve our encoder's alignment with both general visual knowledge and diverse document and web agentic tasks: (i) We introduce a novel model-merging scheme, combining encoders trained with different text decoders to ensure broad compatibility with different web agentic architectures. (ii) We use ensemble training to fuse features from pretrained generalist encoders (e.g., SigLIP2) with our own document and web-specific representations. Extensive experiments on classic document tasks, VQAs, web localization, and agent-based benchmarks validate the effectiveness of our approach, establishing DAVE as a strong vision encoder for document and web applications.

Figures (4)

• Figure 1: DAVE Overview. Stage 1 trains the vision encoder with a decoder using MAE, learning strong structural and spatial priors from unlabeled data. Stage 2 performs autoregressive pretraining on diverse tasks with different text decoders and fuses the high-level semantic features from SigLIP 2. After that, different encoders are combined into a single one by learning a merge coefficient using unsupervised representation alignment, while keeping the encoders frozen.
• Figure 2: Each row indicates an additional modification to the training strategy.
• Figure 3: Inter-patch standard deviation across different data sources.
• Figure 4: Training loss curve of MAE with normalized pixel as objective.