LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?

TL;DR

LiveCodeBench Pro tackles whether LLMs truly match top competitive programmers by providing a live, contamination‑free benchmark sourced from Codeforces, ICPC, and IOI, with expert annotations that diagnose problem categories and cognitive focus. The authors show that frontier models remain limited in nuanced algorithmic reasoning and edge‑case handling, though they can perform well on structured, knowledge‑heavy tasks and improve with pass@k and tool augmentation. Key contributions include a live curation pipeline, fine‑grained taxonomy, and Bayesian Elo‑based human–model comparability, all enabling granular diagnostics rather than a single pass/fail metric. The work reveals a persistent gap to human grandmasters on hard problems while offering diagnostics to steer future improvements in code‑centric LLM reasoning and tool use.

Abstract

Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning.

Figures (7)

• Figure 1: LiveCodeBench Pro leaderboard. Elo rating versus average cost per problem for various models. The gray region zooms in on the cluster of non-reasoning models.
• Figure 2: Tag-wise model performance. The $x$-axis represents different problem types, ranging from knowledge-heavy problems in blue (e.g., segment tree, implementation, data structures) to logic-heavy in red (e.g., combinatorics, dynamic programming, mathematics) to observation-heavy problems in green (e.g., greedy, interactive, game theory). The $y$-axis corresponds to Codeforces-equivalent Elo ratings, with human percentile benchmarks labeled (e.g., Master). Models tend to excel on knowledge-heavy and logic-heavy problems but struggle on observation-heavy ones.
• Figure 3: Failure reasons in treemap. Comparing rejected submissions between o3-mini and humans. Each block represents a specific rejection tag; its size is proportional to the total count of rejections for that tag, with the inscribed text showing the o3-mini:human ratio. The color of the block indicates the contribution rate of o3-mini to that tag's rejections: red signifies a higher proportion from o3-mini, while blue indicates a higher proportion from humans.
• Figure 4: o4-mini performance under pass@k settings. The plot shows the pass rates for Easy and Medium tier problems, and the corresponding Elo rating changes as the number of attempts ($k$) increases. All variants show 0% pass rate on the hard tier in the evaluation.
• Figure 5: Performance improvement with pass@k setting across tags. The Elo ratings for pass@1 and pass@10 across different problem categories show significant performance improvements. The evaluated model is o4-mini-medium. Error bars representing the 95% confidence intervals.