Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models

Shenyu Zheng, Ximing Dong, Xiaoshuang Liu, Gustavo Oliva, Chong Chun Yong, Dayi Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan

TL;DR

Addressing the reliability of Codeforces-based Elo ratings for LLM evaluation, the paper identifies submission ordering, contest selection, and run-to-run variability as hidden biases. It builds a benchmark of 37 Codeforces contests and 13,691 generated test cases to systematically probe these factors across five advanced LLMs, using automated verifiers for correctness. The results show substantial score drift: up to 394 points from submission order, up to 1,122 points from contest divisions, and up to 349 points across runs, with a striking 1,348-point difference observed in a single contest. The study argues that direct Elo comparisons are unreliable without standardized evaluation pipelines and transparent reporting, and it provides concrete recommendations for robust benchmarking and reproducibility.

Abstract

As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming capabilities. However, these ratings are often reported without critical experimental details, leading to significant discrepancies illustrated by recent reports where the score of the same model version fluctuated by nearly 500 points. This paper presents a systematic empirical study on the hidden factors biasing Elo evaluations: (1) the temporal ordering of submissions, (2) contest difficulty selection, and (3) run to run stochastic variability of LLMs. Utilizing a controlled benchmark of 37 recent Codeforces contests and 13,691 generated test cases, we demonstrate that Elo scores are highly sensitive to these parameters. Our findings reveal that varying submission orders can shift scores by 394 points, while contest selection can cause differences of up to 1,122 points for the same model. Run to run performance exhibits substantial instability, with a maximum difference of 349 points in mean scores observed when evaluating identical contests. We conclude that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.

When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models

TL;DR

Addressing the reliability of Codeforces-based Elo ratings for LLM evaluation, the paper identifies submission ordering, contest selection, and run-to-run variability as hidden biases. It builds a benchmark of 37 Codeforces contests and 13,691 generated test cases to systematically probe these factors across five advanced LLMs, using automated verifiers for correctness. The results show substantial score drift: up to 394 points from submission order, up to 1,122 points from contest divisions, and up to 349 points across runs, with a striking 1,348-point difference observed in a single contest. The study argues that direct Elo comparisons are unreliable without standardized evaluation pipelines and transparent reporting, and it provides concrete recommendations for robust benchmarking and reproducibility.

Abstract

As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming capabilities. However, these ratings are often reported without critical experimental details, leading to significant discrepancies illustrated by recent reports where the score of the same model version fluctuated by nearly 500 points. This paper presents a systematic empirical study on the hidden factors biasing Elo evaluations: (1) the temporal ordering of submissions, (2) contest difficulty selection, and (3) run to run stochastic variability of LLMs. Utilizing a controlled benchmark of 37 recent Codeforces contests and 13,691 generated test cases, we demonstrate that Elo scores are highly sensitive to these parameters. Our findings reveal that varying submission orders can shift scores by 394 points, while contest selection can cause differences of up to 1,122 points for the same model. Run to run performance exhibits substantial instability, with a maximum difference of 349 points in mean scores observed when evaluating identical contests. We conclude that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.
Paper Structure (21 sections, 2 equations, 4 figures)

This paper contains 21 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Experiment Design
  4. Benchmark Construction
  5. Problem Collection
  6. Test Case Collection and Generation
  7. Local Model Evaluation
  8. Multiple Solutions Verification
  9. Elo Score Calculation
  10. Statistics of Benchmark
  11. Base LLMs
  12. Approaches of RQs
  13. Approach of RQ1
  14. Approach of RQ2
  15. Approach of RQ3
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Overview of our experimental methodology
  • Figure 2: Delta Elo scores (i.e., optimal-case - worst-case) of evaluated contests for different models along with different numbers of output solutions ($n$). Mean value is marked as a diamond.
  • Figure 3: Elo scores across four divisions for evaluated models. Mean value is marked as a diamond.
  • Figure 4: Elo scores of evaluated contests across three runs for evaluated models. Mean value is marked as a diamond.