Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Elo Rating Reliable? A Study Under Model Misspecification

Shange Tang, Yuanhao Wang, Chi Jin

TL;DR

The paper investigates the reliability of Elo rating when real-world data deviate from the stationary Bradley-Terry model. It shows through likelihood-ratio testing that BT is frequently misspecified and that matchmaking and player skills often evolve over time, yet Elo-based methods often outperform more complex alternatives in predicting outcomes, especially in sparse data regimes. By reframing Elo as online gradient descent within online convex optimization, it establishes no-regret guarantees under misspecification and non-stationarity, and demonstrates that data sparsity fundamentally drives performance trade-offs between simple and complex models. Synthetic and real-world experiments reveal a strong link between predictive accuracy and ranking quality, while also highlighting scenarios where Elo can fail to produce a consistent global ranking under non-uniform matchmaking. Together, these results clarify Elo’s practical strengths and limitations, with implications for model selection and interpretation in competitive settings and AI evaluation tasks.

Abstract

Elo rating, widely used for skill assessment across diverse domains ranging from competitive games to large language models, is often understood as an incremental update algorithm for estimating a stationary Bradley-Terry (BT) model. However, our empirical analysis of practical matching datasets reveals two surprising findings: (1) Most games deviate significantly from the assumptions of the BT model and stationarity, raising questions on the reliability of Elo. (2) Despite these deviations, Elo frequently outperforms more complex rating systems, such as mElo and pairwise models, which are specifically designed to account for non-BT components in the data, particularly in terms of win rate prediction. This paper explains this unexpected phenomenon through three key perspectives: (a) We reinterpret Elo as an instance of online gradient descent, which provides no-regret guarantees even in misspecified and non-stationary settings. (b) Through extensive synthetic experiments on data generated from transitive but non-BT models, such as strongly or weakly stochastic transitive models, we show that the ''sparsity'' of practical matching data is a critical factor behind Elo's superior performance in prediction compared to more complex rating systems. (c) We observe a strong correlation between Elo's predictive accuracy and its ranking performance, further supporting its effectiveness in ranking.

Is Elo Rating Reliable? A Study Under Model Misspecification

TL;DR

The paper investigates the reliability of Elo rating when real-world data deviate from the stationary Bradley-Terry model. It shows through likelihood-ratio testing that BT is frequently misspecified and that matchmaking and player skills often evolve over time, yet Elo-based methods often outperform more complex alternatives in predicting outcomes, especially in sparse data regimes. By reframing Elo as online gradient descent within online convex optimization, it establishes no-regret guarantees under misspecification and non-stationarity, and demonstrates that data sparsity fundamentally drives performance trade-offs between simple and complex models. Synthetic and real-world experiments reveal a strong link between predictive accuracy and ranking quality, while also highlighting scenarios where Elo can fail to produce a consistent global ranking under non-uniform matchmaking. Together, these results clarify Elo’s practical strengths and limitations, with implications for model selection and interpretation in competitive settings and AI evaluation tasks.

Abstract

Elo rating, widely used for skill assessment across diverse domains ranging from competitive games to large language models, is often understood as an incremental update algorithm for estimating a stationary Bradley-Terry (BT) model. However, our empirical analysis of practical matching datasets reveals two surprising findings: (1) Most games deviate significantly from the assumptions of the BT model and stationarity, raising questions on the reliability of Elo. (2) Despite these deviations, Elo frequently outperforms more complex rating systems, such as mElo and pairwise models, which are specifically designed to account for non-BT components in the data, particularly in terms of win rate prediction. This paper explains this unexpected phenomenon through three key perspectives: (a) We reinterpret Elo as an instance of online gradient descent, which provides no-regret guarantees even in misspecified and non-stationary settings. (b) Through extensive synthetic experiments on data generated from transitive but non-BT models, such as strongly or weakly stochastic transitive models, we show that the ''sparsity'' of practical matching data is a critical factor behind Elo's superior performance in prediction compared to more complex rating systems. (c) We observe a strong correlation between Elo's predictive accuracy and its ranking performance, further supporting its effectiveness in ranking.

Paper Structure

This paper contains 70 sections, 3 theorems, 37 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods for rating game players
  4. Analysis of Elo score
  5. Misspecification of Bradley-Terry model
  6. Learning to rank
  7. Preliminary
  8. Algorithms
  9. Elo rating:
  10. Glicko, TrueSkill --- "Elo-like" rating:
  11. Elo2k, Pairwise --- more complexity rating systems:
  12. Datasets
  13. Experiments on Real-world Matching Data
  14. Real-world games are neither BT nor stationary
  15. Rejecting BT on real-world dataset
  16. ...and 55 more sections

Key Result

Theorem 4.1

For convex cost functions $\{f_t\}_{t=1}^{T}$ and convex set $\mathcal{K}$, online gradient descent with step sizes $\{\eta_t = \frac{D}{G\sqrt{t}}\}$ guarantee the following for all $T>1$: where $D$ is the upper bound on the diameter of $\mathcal{K}$, and $G$ is an upper bound on the norm of the subgradients of $f_t$ over $\mathcal{K}$.

Figures (5)

  • Figure 1: Elo and Elo2k's prediction performance in sparse datasets.
  • Figure 2: Ranking performance correlated with prediction.
  • Figure 3: Matchmaking in chess dataset.L: scatter plot of Elo score of the two players for each game, down-sampled for clarity; R: histogram for the percentile ranking difference of two players.
  • Figure 4: Elo score vs. bootstrap Elo scores in chess. Left: cosine similarity to the mean of $\theta^{\rm bootstrap}$; Right: visualization of $\theta^{\rm elo}$ vs. $\theta^{\rm bootstrap}$ via SVD for $\eta=0.02$.
  • Figure 5: In real datasets, sparsity strongly influences prediction. Elo, TrueSkill, Glicko achieves the best prediction in sparse datasets, while Elo2k and Pairwise outperforms Elo and its variants in dense datasets.

Theorems & Definitions (7)

  • Theorem 4.1
  • Definition 1: SST
  • Definition 2: WST
  • Theorem 4.2
  • Theorem G.1
  • proof
  • Example 1