Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

TL;DR

RankLLM addresses the limitation of traditional LLM benchmarks by explicitly modeling question difficulty and model competency. It uses a non-parametric, bidirectional score-propagation framework on a directed bipartite graph to compute stationary difficulty $\pi_q$ and competency $\pi_m$ through damped iterations, converging to a unique solution. Empirical validation across six benchmarks with 30 models (35,550 questions) shows strong alignment with human difficulty judgments (≈90%) and superiority over IRT baselines, while remaining scalable and robust to model pool and dataset perturbations. The method provides finer-grained rankings, revealing strengths on hard items and informing dataset design, evaluation strategies, and potential educational and benchmarking applications.

Abstract

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

TL;DR

RankLLM addresses the limitation of traditional LLM benchmarks by explicitly modeling question difficulty and model competency. It uses a non-parametric, bidirectional score-propagation framework on a directed bipartite graph to compute stationary difficulty and competency through damped iterations, converging to a unique solution. Empirical validation across six benchmarks with 30 models (35,550 questions) shows strong alignment with human difficulty judgments (≈90%) and superiority over IRT baselines, while remaining scalable and robust to model pool and dataset perturbations. The method provides finer-grained rankings, revealing strengths on hard items and informing dataset design, evaluation strategies, and potential educational and benchmarking applications.

Abstract

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.
Paper Structure (40 sections, 1 theorem, 11 equations, 13 figures, 16 tables)

This paper contains 40 sections, 1 theorem, 11 equations, 13 figures, 16 tables.

Table of Contents

  1. Introduction
  2. RankLLM
  3. Score in RankLLM
  4. Directed Bipartite Graph and Performance Matrices
  5. Propagating Scores via Iterative Refinement
  6. Extending RankLLM to Continuous Scores
  7. Experiment
  8. Experiment Setup
  9. Main Results
  10. Human Evaluation
  11. Efficiency and Scalability Analysis of RankLLM
  12. Robustness and Extensibility Analysis of RankLLM
  13. Case Study: Validating Difficulty-Based Evaluation
  14. Conclusion
  15. Appendix
  16. ...and 25 more sections

Key Result

Theorem C.1

Let $P_{Q \to M} \in \mathbb{R}^{Q \times M}$ and $P_{M \to Q} \in \mathbb{R}^{M \times Q}$ be the row-stochastic matrices defined in subsec:transition_matrices. For any damping factor $\alpha \in (0,1)$, the iterative process defined in eq:iter_q and eq:iter_m converges to a unique stationary distr Furthermore, $\pi_Q$ and $\pi_M$ have strictly positive entries.

Figures (13)

  • Figure 1: Schematic of RankLLM’s Weighted Ranking Pipeline.
  • Figure 2: Detailed process demonstration of score propagation in RankLLM, which includes Evaluation Inputs, Bipartite Graph & Matrices, and Iterative Score Propagation.
  • Figure 3: RankLLM scores and accuracies of models, both normalized to a fixed maximum of 100 to facilitate direct comparison. The full names of the models are provided in \ref{['app:model_details']}.
  • Figure 4: Relationship between model performance and parameter scale.
  • Figure 5: The proportion of Easy/Medium/Hard questions within correctly answered samples across Llama/Yi variants.
  • ...and 8 more figures

Theorems & Definitions (2)

  • Theorem C.1: Existence and Uniqueness of the Stationary Distribution
  • proof : Proof sketch of \ref{['thm:existence-uniqueness']}