VisionArena: 230K Real World User-VLM Conversations with Preference Labels

TL;DR

VisionArena tackles the need for realistic, open-ended vision-language benchmarks by compiling 230K real-world user–VLM conversations from Chatbot Arena, spanning 138 languages and 45 models across three subsets: VisionArena-Chat, VisionArena-Battle, and VisionArena-Bench. The work introduces a unified crowd-driven evaluation framework, using a Bradley–Terry model to derive arena scores from user preferences, and analyzes how factors like response length and formatting influence judgments. It demonstrates the utility of VisionArena for instruction tuning, showing significant performance gains over baseline instruction-tuning datasets, and proposes VisionArena-Bench as a cost-effective offline proxy for live leaderboard rankings. The paper discusses limitations in coverage and language diversity, outlines moderation and privacy safeguards, and offers the dataset and baseline models to accelerate future research in aligning VLMs with human preferences.

Abstract

With the growing adoption and capabilities of vision-language models (VLMs) comes the need for benchmarks that capture authentic user-VLM interactions. In response, we create VisionArena, a dataset of 230K real-world conversations between users and VLMs. Collected from Chatbot Arena - an open-source platform where users interact with VLMs and submit preference votes - VisionArena spans 73K unique users, 45 VLMs, and 138 languages. Our dataset contains three subsets: VisionArena-Chat, 200k single and multi-turn conversations between a user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark of 500 diverse user prompts that efficiently approximate the live Chatbot Arena model rankings. Additionally, we highlight the types of question asked by users, the influence of response style on preference, and areas where models often fail. We find open-ended tasks like captioning and humor are highly style-dependent, and current VLMs struggle with spatial reasoning and planning tasks. Lastly, we show finetuning the same base model on VisionArena-Chat outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai

Figures (28)

• Figure 1: Samples from VisionArena Conversations. VisionArena contains conversations from real users covering a variety of domains.
• Figure 2: Bootstrap B.T. model scores for VisionArena-Battle. Proprietary models like Gemini 1.5 Pro and GPT-4o are at the top of the leaderboard, with open models like Llava 1.6, MiniCPM, CogVLMv2, and Phi3 obtaining the lowest ratings. InternVL2 is the highest rated open model, although as shown in Section \ref{['sec:style_analysis']}, this is largely due to response style rather than model capability.
• Figure 3: Comparison of top 5 topic clusters between WildVision-Chat and VisionArena-Chat. Compared to WildVision, the most popular topics clusters in VisionArena capture more real world tasks, specifically in STEM fields.
• Figure 4: Descriptions of VisionArena categories.
• Figure 5: Category Distribution. Excluding preset examples. We see that direct chat data contains a higher proportion of coding, homework, and diagram questions while battle data contains more captioning, humor, and creative writing questions.