Search Arena: Analyzing Search-Augmented LLMs

TL;DR

Search Arena introduces the first large-scale, crowd-sourced dataset of real-world, multi-turn interactions with search-augmented LLMs and accompanying human preferences. By analyzing prompts, intents, and citation behaviors, the work reveals how citation quantity and source type shape user judgments, and shows that web search generally helps in information-seeking tasks while relying solely on parametric knowledge can hurt in search-heavy contexts. The cross-arena evaluation demonstrates that search augmentation improves performance in information retrieval and synthesis settings, but is less beneficial when the task relies on closed-book reasoning. The dataset and accompanying code enable rigorous, human-centered evaluation of search-enabled LLMs and set the stage for improved trust and grounding in AI systems.

Abstract

Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static encyclopedic sources are not always appropriate and reliable. To assess performance across different settings, we conduct cross-arena analyses by testing search-augmented LLMs in a general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research in this direction. Our dataset and code are available at: https://github.com/lmarena/search-arena.

Figures (28)

• Figure 1: (Left) Nine intent categories with representative examples (truncated). In-the-wild user prompts are often ambiguous and require real-time web retrieval. (Right) Distribution of intents across user prompts. The majority of queries require more than a simple factual lookup and range from information synthesis to creative content generation. The Other category is excluded from the visualizations.
• Figure 2: (Left) Search Arena prompt language distribution. The dataset is multilingual, spanning over 70 languages, with English prompts accounting for 58.3% of the data. (Right) Prompt length distribution of Search Arena (blue), BrowseComp (purple), and SimpleQA (green). Search Arena prompt lengths are more spread out and cover the range of BrowseComp browsecomp and SimpleQA simpleqa questions.
• Figure 3: (Left) Reasoning trace example, containing multi-document analysis, filtering, and synthesis. (Right) Example of a rejected response citing Wikipedia for a sports news question. The preferred response cited the sports division of a news outlet, containing more up-to-date information.
• Figure 4: (Left) Positive relationship between model score and average response length. (Right) Positive relationship between model score and average number of citations.
• Figure 5: (Left) Response length distribution across user intent categories. Responses to Factual Lookup prompts are more concise (168.3 words on average) compared to other categories. (Right) Citation count distribution across user intent categories. Responses to Recommendation (6.9 on average) and Info Synthesis (6.8 on average) prompts contain more citations compared to Factual Lookup (5.7) and Text Processing (4.2) prompts.