Self-Improving VLM Judges Without Human Annotations

TL;DR

The paper tackles the high cost and rapid obsolescence of human preference data for training Vision-Language Model judges. It introduces a three-stage, self-training loop that generates synthetic preference data, filters judgments with reasoning traces from the current judge, and fine-tunes the judge on filtered signals, iterating to improve performance. Empirically, an 11B Llama-3.2-11B-Vision-Instruct judge achieves competitive VL-RewardBench results (0.51) and outperforms larger models on several dimensions, notably general instruction following and hallucination detection. These results demonstrate a viable path toward self-improving, annotation-free judges that can co-evolve with advancing VLMs and adapt to new visual domains without ground-truth labels.

Abstract

Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.

Figures (7)

• Figure 1: Self-improving VLM judge: iterative synthetic preference data generation and judge model fine-tuning pipeline. I. Synthetic preference pair generation. We create synthetic preference pairs tailored to different question types. For open solution space questions (long answers, captions), we generate an original response and deliberately inject meaningful errors to create a less preferred version. For closed solution space questions (numerical, multiple choice, short phrases), we generate multiple candidates and pair the majority answer with a random alternative. II. Iterative judge training data generation. We use the previous-iteration judge model to evaluate the newly synthesized preference pairs and gather the judge's reasoning traces. We retain only judgments that align with our synthetic preferences. III. Judge model training. We fine-tune the previous-iteration judge model on these filtered reasoning traces. We iterate this three-step process several times. More details in §\ref{['sec:method']}.
• Figure 2: Judge model performance across training iterations. The left lanel shows average VLRB scores and the right panel shows average MMRB scores. After 4 iterations, our 11B judge model is comparable with Claude-3.5 and Llama-90B on VLRB.
• Figure 3: Increasing % of data sampled from each training iteration
• Figure 4: Performance comparing using majority voting and correct answer to filter synthetic pairs before sampling. For VLRB reasoning and MMRB VQA using majority voting to filter the synthetic pairs yields better performance after a few iterations. When reducing data size to the same with correct answer filtering, majority performance also reduced.
• Figure 5: The dimensions that showed the most significant improvements (VLRB General, Hallucination) and the least significant improvements (MMRB Safety, General). More details in §\ref{['sec:analysis_bytask']}.