ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMs

TL;DR

ReLE tackles benchmark saturation in Chinese LLM evaluation by diagnosing capability anisotropy through a structured Domain x Capability matrix and a scalable live evaluation pipeline. It combines Symbolic-Grounded Hybrid Scoring with a Dynamic Variance-Aware Scheduler to cut evaluation costs by about 70% while preserving ranking fidelity (ρ = $0.96$) and exposing substantial ranking instability (RSA ≈ $11.4$) that traditional benchmarks conceal. The framework leverages a 207,843-sample benchmark across 7 domains and 22 capabilities, enabling fine-grained insights into domain knowledge versus reasoning ability and tool-use skills, with a robust decontamination and latency protocol. Practically, ReLE offers a high-frequency diagnostic monitor that informs multi-objective model selection, cost-performance trade-offs, and enterprise deployment strategies, while providing a path toward safety, process-aware reasoning evaluation, and collaborative evaluation workflows.

Abstract

Large Language Models (LLMs) have achieved rapid progress in Chinese language understanding, yet accurately evaluating their capabilities remains challenged by benchmark saturation and prohibitive computational costs. While static leaderboards provide snapshot rankings, they often mask the structural trade-offs between capabilities. In this work, we present ReLE (Robust Efficient Live Evaluation), a scalable system designed to diagnose Capability Anisotropy, the non-uniformity of model performance across domains. Using ReLE, we evaluate 304 models (189 commercial, 115 open-source) across a Domain $\times$ Capability orthogonal matrix comprising 207,843 samples. We introduce two methodological contributions to address current evaluation pitfalls: (1) A Symbolic-Grounded Hybrid Scoring Mechanism that eliminates embedding-based false positives in reasoning tasks; (2) A Dynamic Variance-Aware Scheduler based on Neyman allocation with noise correction, which reduces compute costs by 70\% compared to full-pass evaluations while maintaining a ranking correlation of $ρ=0.96$. Our analysis reveals that aggregate rankings are highly sensitive to weighting schemes: models exhibit a Rank Stability Amplitude (RSA) of 11.4 in ReLE versus $\sim$5.0 in traditional benchmarks, confirming that modern models are highly specialized rather than generally superior. We position ReLE not as a replacement for comprehensive static benchmarks, but as a high-frequency diagnostic monitor for the evolving model landscape.

Figures (4)

• Figure 1: ReLE System Architecture. The system decouples models, tasks, and scoring. Layout (left to right): (1) Model Interface Layer: Unifies commercial and open-source models; (2) Unified Prompt Schema: Standardizes inputs across tasks; (3) Task Scheduler: Manages cost-aware execution; (4) Scoring Pipeline: Normalizes diverse outputs; (5) Leaderboard Module: Updates dynamic rankings.
• Figure 2: Capability Radar Plot and Anisotropy Visualization. By comparing representative models across dimensions (Language, Reasoning, Education, etc.), we observe highly irregular shapes rather than circular profiles. The distinct, non-overlapping areas of these radar charts across different model families quantify the divergence in capability focus, demonstrating that LLM capability is highly anisotropic.
• Figure 3: Ranking Instability under Capability Reweighting. Tracking individual models across schemes (General-heavy, Professional-heavy, Reasoning-heavy) reveals large rank fluctuations, indicating leaderboard sensitivity to subjective weights.
• Figure 4: Failure Pattern Distribution across Capability Dimensions. Failure patterns (Hallucination, Reasoning Error, etc.) are domain-specific rather than size-dependent. Professional domains exhibit higher factual hallucination rates.