LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics

TL;DR

The paper tackles the challenge of evaluating Large Language Models across diverse benchmarks by proposing Competitive Swiss-System Dynamics (CSD), a dynamic, multi-round contest that captures path-dependent performance under competitive pressure. It introduces the Pairwise Win-rate Tensor $W$, Swiss-System pairing, and a structured elimination mechanism to produce a statistically robust Expected Win Score $\hat{E}[S_m]$ via Monte Carlo simulation with $N=100{,}000$ iterations, while a Failure Sensitivity Analysis using elimination levels $T_k$ profiles robustness and risk. The approach yields a nuanced, deployment-relevant ranking that accounts for sequential task dependencies and penalizes fragility, demonstrated through experiments on 29 LLMs across 38 benchmarks. The work offers a principled, risk-informed alternative to static aggregation, with practical implications for model selection in complex, multi-stage workflows and potential extensions to agentic performance prediction and weighted-dataset evaluation.

Abstract

The rapid proliferation of Large Language Models (LLMs) and diverse specialized benchmarks necessitates a shift from fragmented, task-specific metrics to a holistic, competitive ranking system that effectively aggregates performance across multiple ability dimensions. Primarily using static scoring, current evaluation methods are fundamentally limited. They struggle to determine the proper mix ratio across diverse benchmarks, and critically, they fail to capture a model's dynamic competitive fitness or its vulnerability when confronted with sequential, high-stakes tasks. To address this, we introduce the novel Competitive Swiss-System Dynamics (CSD) framework. CSD simulates a multi-round, sequential contest where models are dynamically paired across a curated sequence of benchmarks based on their accumulated win-loss record. And Monte Carlo Simulation ($N=100,000$ iterations) is used to approximate the statistically robust Expected Win Score ($E[S_m]$), which eliminates the noise of random pairing and early-round luck. Furthermore, we implement a Failure Sensitivity Analysis by parameterizing the per-round elimination quantity ($T_k$), which allows us to profile models based on their risk appetite--distinguishing between robust generalists and aggressive specialists. We demonstrate that CSD provides a more nuanced and context-aware ranking than traditional aggregate scoring and static pairwise models, representing a vital step towards risk-informed, next-generation LLM evaluation.

Figures (6)

• Figure 1: Overall ranking of 29 advanced LLMs across 38 recent widely-used and open-sourced benchmarks given by our Competitive Swiss-System Dynamics framework. Check if this aligns with your insights.
• Figure 2: The Illusion of Static Aggregation. (A) Averaging scores hides foundational failures. (B) In a realistic sequential workflow, failure in a foundational task (Step 1) blocks downstream capabilities (Step 2), illustrating the path dependency of model performance.
• Figure 3: Overall ranking of 29 advanced LLMs across 38 recent widely-used and open-sourced benchmarks given by our CSD framework, highlighting the models organized into four tiers.
• Figure 4: Base Performance and Sensitivity Coefficient of 29 Models. The base performance is the average model score given by our CSD framework when there is no model elimination ($T_k=0$). The sensitivity coefficient is the gradient in average score when $T_k$ increases from 0 to 2, calculated as $(E[S_m|T_k=2] - E[S_m|T_k=0])/2$. A more negative sensitivity coefficient indicates greater model shortcomings or susceptibility to elimination.
• Figure 5: Score Perturbatioin Analysis on Extremely Low Scores.

Theorems & Definitions (3)

• Example 1: Sensitivity to Zero Scores on IFEval and MulDimIF
• Example 2: Sensitivity to Zero Scores on Four Benchmarks
• Example 3: Web Navigation and Data Extraction