K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge

TL;DR

The paper tackles the challenge of scalable, human-aligned evaluation for rapidly evolving visual generative models, where traditional metrics and static benchmarks fail to reflect real user preferences. It introduces K-Sort Eval, a VLM-as-a-Judge framework that combines posterior correction for observation noise and a dynamic matching strategy to guide ($K$+1)-wise comparisons using a high-quality dataset derived from K-Sort Arena. Key contributions include a data-curation pipeline with Spearman-based filtering and safety screening, a noise-aware Bayesian updating mechanism with adaptive weighting via $\lambda'$, and an uncertainty-diversity driven instance selection scheme, all integrated into an efficient evaluation loop that stops when $\sigma$ falls below a threshold and yields $S = \mu - \eta\sigma$. Empirical results show strong agreement with human Arena judgments while requiring far fewer model runs (typically $<90$), and the approach remains effective across images, videos, and compressed-model settings, offering a practical, scalable path for ongoing visual-generation evaluation.

Abstract

The rapid development of visual generative models raises the need for more scalable and human-aligned evaluation methods. While the crowdsourced Arena platforms offer human preference assessments by collecting human votes, they are costly and time-consuming, inherently limiting their scalability. Leveraging vision-language model (VLMs) as substitutes for manual judgments presents a promising solution. However, the inherent hallucinations and biases of VLMs hinder alignment with human preferences, thus compromising evaluation reliability. Additionally, the static evaluation approach lead to low efficiency. In this paper, we propose K-Sort Eval, a reliable and efficient VLM-based evaluation framework that integrates posterior correction and dynamic matching. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, with each instance containing the outputs and rankings of K models. When evaluating a new model, it undergoes (K+1)-wise free-for-all comparisons with existing models, and the VLM provide the rankings. To enhance alignment and reliability, we propose a posterior correction method, which adaptively corrects the posterior probability in Bayesian updating based on the consistency between the VLM prediction and human supervision. Moreover, we propose a dynamic matching strategy, which balances uncertainty and diversity to maximize the expected benefit of each comparison, thus ensuring more efficient evaluation. Extensive experiments show that K-Sort Eval delivers evaluation results consistent with K-Sort Arena, typically requiring fewer than 90 model runs, demonstrating both its efficiency and reliability.

Figures (5)

• Figure 1: Overview of the proposed K-Sort Eval. First, a high-quality dataset is curated through consistency filtering. When evaluating a new model, we begin with dynamic matching to select the most informative instance. Then, two prompt strategies are employed to effectively guide the VLM and mitigate hallucinations. Finally, Bayesian updating with correction is performed, where the discrepancy between VLM prediction and dataset supervision is treated as observation noise to correct the posterior estimation of model capability.
• Figure 2: Visualization of the evaluation processes. With posterior correction, K-Sort Eval achieves a smoother trajectory and produces more accurate results that are consistent with K-Sort Arena.
• Figure 3: Number of runs required for the new model in the evaluation.
• Figure 4: Correlations of different methods with actual human preferences.
• Figure 5: Prompt design that provides voting criteria consistent with human voting in K-Sort Arena, serving as guidance for the VLM Judgement and helping reduce hallucinations.

Theorems & Definitions (2)

• Lemma 1
• proof