K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences

TL;DR

K-Sort Arena tackles the inefficiency and noise-proneness of traditional Arena-based human evaluations for visual generation by introducing K-wise comparisons (K>2) that yield richer information per round. The approach combines probabilistic capability modeling with Bayesian updating and a UCB-based exploration–exploitation matchmaking strategy, enabling faster and more robust ranking than pairwise methods such as ELO. Empirical results from simulations show $16.3\times$ faster convergence and improved noise robustness, while the platform supports crowdsourced, real-time leaderboard updates across text-to-image and text-to-video tasks. The open-source platform on Huggingface Space facilitates continuous evaluation and rapid incorporation of new models, offering practical impact for researchers and practitioners alike.

Abstract

The rapid advancement of visual generative models necessitates efficient and reliable evaluation methods. Arena platform, which gathers user votes on model comparisons, can rank models with human preferences. However, traditional Arena methods, while established, require an excessive number of comparisons for ranking to converge and are vulnerable to preference noise in voting, suggesting the need for better approaches tailored to contemporary evaluation challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable platform based on a key insight: images and videos possess higher perceptual intuitiveness than texts, enabling rapid evaluation of multiple samples simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing K models to engage in free-for-all competitions, which yield much richer information than pairwise comparisons. To enhance the robustness of the system, we leverage probabilistic modeling and Bayesian updating techniques. We propose an exploration-exploitation-based matchmaking strategy to facilitate more informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster convergence compared to the widely used ELO algorithm. To further validate the superiority and obtain a comprehensive leaderboard, we collect human feedback via crowdsourced evaluations of numerous cutting-edge text-to-image and text-to-video models. Thanks to its high efficiency, K-Sort Arena can continuously incorporate emerging models and update the leaderboard with minimal votes. Our project has undergone several months of internal testing and is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena

Figures (11)

• Figure 1: Comparison between K-Sort Arena and Chatbot Arena chiang2024chatbot. K-Sort Arena employs K-wise comparisons (K$>$2) to get richer information from user votes. Notably, it introduces probabilistic modeling and an effective matchmaking strategy, significantly improving efficiency and reliability.
• Figure 2: Overview of the proposed K-Sort Arena. K-wise comparisons (K$>$2) and the advanced matching strategy can significantly accelerate ranking convergence, achieving stable ranking with minimal user votes. Probabilistic modeling and Bayesian updating can enhance the robustness of model capability representation, thus resulting in greater efficiency and reliability.
• Figure 3: Comparison of numerical modeling (ELO elo1967elo) and probabilistic modeling (ours) at K=2, separately with and without preference noise. Probabilistic modeling can converge quickly, while numerical modeling stays oscillating and fails to converge.
• Figure 4: Comparison of different K values when applying UCB matchmaking. K$\in$[2,4,6]. As K increases, the convergence becomes faster and more stable.
• Figure 5: Comparison of different matchmaking strategies at K=4, including random (ELO elo1967elo), Skill (TrueSkill herbrich2006trueskill), and UCB (ours). The proposed exploitation-exploration based strategy achieves the fastest convergence.