Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models

TL;DR

Doc-PP introduces a new benchmark to evaluate policy preservation in multimodal document QA, using real-world PDFs paired with explicit non-disclosure policies that require cross-modal reasoning to identify and redact sensitive content. The study reveals a reasoning-induced safety gap and an OCR paradox, where enhanced perception and cross-modal reasoning can inadvertently increase information leakage. To address these vulnerabilities, the authors propose Decompose-Verify-Aggregate (DVA), a lightweight framework that decouples information synthesis from policy verification and significantly reduces leakage across document types and query settings. Collectively, Doc-PP and DVA establish a practical baseline for policy-compliant document understanding in large vision-language models, with implications for safer deployment in real-world document QA tasks.

Abstract

The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering is often constrained by dynamic, user-defined policies that dictate information disclosure based on context. While ensuring adherence to these explicit constraints is critical, existing safety research primarily focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents. In this paper, we introduce Doc-PP (Document Policy Preservation Benchmark), a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies. Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities, effectively circumventing existing safety constraints. Furthermore, we identify that providing extracted text improves perception but inadvertently facilitates leakage. To address these vulnerabilities, we propose DVA (Decompose-Verify-Aggregation), a structural inference framework that decouples reasoning from policy verification. Experimental results demonstrate that DVA significantly outperforms standard prompting defenses, offering a robust baseline for policy-compliant document understanding

Figures (17)

• Figure 1: An example of multimodal policy preservation failure. The LVLM leaks protected information by reasoning across visual and textual evidence, violating an explicit non-disclosure policy. Red text indicates the leaked information.
• Figure 2: Overview of the three-stage Doc-PP benchmark construction pipeline: (1) Policy Construction, where confidential targets are generated and filtered; (2) Query Construction, producing reasoning-heavy queries; and (3) Evaluation, using a checklist-based framework to measure leakage and faithfulness.
• Figure 3: Correlation between document type, text ratio, and information leakage.
• Figure 4: Case study comparing various mitigation strategies. While CoT and Revision fail to withhold the restricted value (37% rating), DVA successfully prevents leakage.
• Figure 5: The full prompt used for the policy generation task, including the reasoning operator catalog.