SCAN: Structured Capability Assessment and Navigation for LLMs

TL;DR

SCAN addresses the problem that existing automatic evaluation benchmarks primarily produce model rankings rather than functional understanding by offering a structured framework to characterize LLM capabilities in a fine-grained, domain-spanning manner.Its core approach combines Taxonomy construction via TaxBuilder, data generation via RealMix, visualization/analysis tooling, and a $PC^2$-based pointwise evaluation method to deliver accurate, scalable judgments.The framework is validated across 21 mainstream LLMs, with in-depth analyses of the GPT-OSS family revealing substantial intra-category variability and language-specific strengths/weaknesses that holistic scores miss.SCAN demonstrates practical impact by enabling rapid, interpretable diagnostics for model development, data curation, and targeted improvement, while remaining extensible to future modalities and evaluation dimensions.

Abstract

Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives to human evaluation. While existing research has focused on approximating model rankings, such benchmarks fail to provide users and developers with a comprehensive and fine-grained understanding of a specific model's capabilities. To fill this gap, we propose \textbf{SCAN} (Structured Capability Assessment and Navigation), a practical framework that enables detailed characterization of LLM capabilities through comprehensive and fine-grained evaluation. SCAN incorporates four key components: (1) TaxBuilder, which extracts capability-indicating tags from extensive queries to construct a hierarchical taxonomy automatically; (2) RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag; (3) a suite of visualization and analysis tools that facilitate efficient navigation and analysis of model capabilities; and (4) a PC$^2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach that achieves significantly higher accuracy compared to classic LLM-as-a-Judge method. Using SCAN, we conduct a comprehensive evaluation of 21 mainstream LLMs. Our detailed analysis of the GPT-OSS family reveals substantial performance variations, even within sub-capabilities belonging to the same category of capability. This finding highlights the importance of fine-grained evaluation in accurately understanding LLM behavior. Project homepage and resources are available at \href{https://liudan193.github.io/Feedbacker/}{https://liudan193.github.io/Feedbacker/}.

Figures (34)

• Figure 1: An overview of SCAN framework.
• Figure 1: Statistics of SCAN-D-V0.
• Figure 2: A subset of SCAN-T-V0. Please refer to project page for complete taxonomy.
• Figure 3: An overview of TaxBuilder, a tree-based automatic taxonomy generation.
• Figure 4: Comparison of RealMix and real user queries, judged by five human evaluators.