VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding

TL;DR

VERSE provides a systematic pipeline to analyze and improve Vision-Language Models for Visually-rich Document Understanding by reducing and exploring visual embeddings. By projecting high-dimensional visual embeddings into a Reduced Embedding Space (RES) and clustering them, VERSE identifies error-prone regions and guides targeted synthetic-data augmentation to boost $F_1$ without harming generalization. The approach demonstrates that on-premise models (e.g., Donut, Idefics2) can match or surpass leading SaaS solutions when optimized with RES-aware data strategies, and that macro-structural document features predominantly shape embedding structure. This work offers practical benefits for model feasibility assessment, explainability, and privacy-preserving deployment in VrDU tasks, with future directions toward latent-space data generation and reconstruction.

Abstract

This work introduces VERSE, a methodology for analyzing and improving Vision-Language Models applied to Visually-rich Document Understanding by exploring their visual embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.

Figures (16)

• Figure 1: Paradigm shift proposed in this work. Traditionally, the quality of synthetic images in a dataset is assessed from an anthropocentric perspective, answering the question of whether such images appear photorealistic. In contrast, this work proposes evaluating the images from the model’s perspective, which entails analyzing their visual embeddings to determine whether a synthetic image lies within the target distribution, as perceived by the model itself.
• Figure 2: VERSE methodology components. The validation dataset MERIT secret (A) is processed by models' visual encoders (B) to obtain visual embeddings. F1 scores are obtained after inference on the validation dataset with fine-tuned models on the different MERIT Dataset versions. High-dimensional visual embeddings are reduced to a lower-dimensional space (D), known as the Reduced Embedding Space. This space provides better model interpretability, while overlaying the samples' visual features and the F1 scores enhances model explainability (E). Sections \ref{['subsec:verse_blocks']}.\ref{['subsec:test-dev_dataset']} to \ref{['subsec:verse_blocks']}.\ref{['subsec:clustering']} explain in further detail the components (A-E) involved in the methodology.
• Figure 3: Validation dataset: MERIT Secret, a dataset with real and anonymized samples, with 10 categories subdivided by school. Table \ref{['tab:metadata-condensed']} shows the relevant features extracted in MERIT Secret.
• Figure 4: Training samples used. We employ the Spanish-language subsets of the MERIT Dataset, across its different versions (A). These versions are detailed in Table \ref{['tab:training_dataset_versions']}). Each version comprises data from seven different schools (B). New versions complement the vanilla MERIT Dataset, composed of digital document samples (C) and their renderized versions (D).
• Figure 5: Qualitative illustration of the pipeline used to obtain a single visual embedding per image. The procedure is encoder-specific: Donut, PaliGemma, and LLaVA employ a lightweight MLP projection to align visual and textual embedding dimensions, whereas Idefics2 incorporates a more advanced projection mechanism within the vision encoder. For consistency across models, we extract visual embeddings from the output of the vision block in all cases.