VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data

TL;DR

VERITAS tackles the challenge of noisy SFT data for large multimodal models by grounding critiques in vision priors, ensembling multiple expert LMM critics with domain-aware shrinkage fusion to produce a high-confidence ground-truth score $\hat{S}$, and distilling this into a lightweight GRPO-based critic. The pipeline then generates multiple rewritten answers conditioned on priors and rationales, with the GRPO critic selecting the best candidate, yielding a refined, confidence-graded dataset. Empirical results show substantial downstream gains across six benchmarks, notably +$7.4$ average accuracy for a 7B model and strong alignment with human judgments (Kendall’s $\tau \approx 0.71$) at a fraction of the cost of larger LMMs. The work demonstrates the effectiveness of vision priors, multi-expert evaluation, and principled fusion for robust multimodal data curation, and provides publicly released data and checkpoints to advance research.

Abstract

The quality of supervised fine-tuning (SFT) data is crucial for the performance of large multimodal models (LMMs), yet current data enhancement methods often suffer from factual errors and hallucinations due to inadequate visual perception. To address this challenge, we propose VERITAS, a pipeline that systematically integrates vision priors and multiple state-of-the-art LMMs with statistical methods to enhance SFT data quality. VERITAS leverages visual recognition models (RAM++) and OCR systems (PP-OCRv4) to extract structured vision priors, which are combined with images, questions, and answers. Three LMMs (GPT-4o, Gemini-2.5-Pro, Doubao-1.5-pro) evaluate the original answers, providing critique rationales and scores that are statistically fused into a high-confidence consensus score serving as ground truth. Using this consensus, we train a lightweight critic model via Group Relative Policy Optimization (GRPO), enhancing reasoning capabilities efficiently. Each LMM then refines the original answers based on the critiques, generating new candidate answers; we select the highest-scoring one as the final refined answer. Experiments across six multimodal benchmarks demonstrate that models fine-tuned with data processed by VERITAS consistently outperform those using raw data, particularly in text-rich and fine-grained reasoning tasks. Our critic model exhibits enhanced capability comparable to state-of-the-art LMMs while being significantly more efficient. We release our pipeline, datasets, and model checkpoints to advance research in multimodal data optimization.

Figures (5)

• Figure 1: Performance comparison of different models on various benchmarks.
• Figure 2: Overview of the proposed pipeline VERITAS containing four stages: (1) Vision-Prior Extraction, (2) Stage2: Tri-Expert Assessment with Shrinkage Fusion, (3) Integration with GRPO Critic, (4) Self-Refinement and Answer Selection.
• Figure 3: A Comparative Analysis of Critique Before and After the Integration of Vision Prior in GPT-4o. The sections marked in red denote inaccuracies in the answer and also reflect the critique model's identification of these errors, while the sections highlighted in green represent accurate critique.
• Figure 4: Score distributions from individual critics and after fusion. The fusion results in a more balanced distribution.
• Figure 5: Critic weights before (raw) and after fusion across seven data sources. The fused weights adaptively adjust each critic's influence per domain.