AutoCode: LLMs as Problem Setters for Competitive Programming

TL;DR

The paper tackles the challenge of automatically generating rigorous competitive-programming problems and test data. It presents AutoCode, a closed-loop system that combines a Validator-Generator-Checker framework with an Interactor and a dual-verification protocol to produce and certify both problem statements and their test inputs. Empirical results on large benchmarks show AutoCode achieving high agreement with official judgments and substantially reducing false positives/negatives compared with prior baselines, while also enabling generation of novel, contest-grade problems vetted by experts. The work also provides insights into the strengths and limitations of LLM-driven problem authoring, highlighting reliance on human judgment for quality assessment and outlining directions for scalable self-improvement and more novel problem design.

Abstract

Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.

Figures (4)

• Figure 1: AutoCode introduces a closed-loop multi-role Validator-Generator-Checker framework that enables robust test case generation and scalable, self-verified problem generation for competitive programming. It achieves 98.7% consistency with official judgments. This framework precisely mirrors the process human experts follow when creating programming contest problems.
• Figure 2: Distribution of Non-Consistent Verdicts. Each problem in the 720-problem benchmark is evaluated using 33 submissions generated by different LLMs. The verdicts for 85.6% of the problems are consistent with the official judgments, and 97.1% have three or fewer inconsistencies.
• Figure 3: AutoCode is capable of automatically generating novel competitive programming problems. After careful examination and filtering by multiple human experts, 61.6% achieve a quality level suitable for large language model training, and 3.2% are considered good enough to serve as problems for ICPC/IOI-level contests. Level 1 and 2 correspond to automatic correctness checks performed by the system, while level 3 through 6 involve human evaluation to differentiate among finer levels. It is noted that only the Level 1 questions are wrong, while the Level 2 -- 6 questions are all correct and usable, with the only difference being their quality. Detailed grading rubrics are in Appendix \ref{['appendix:grade']}.
• Figure 4: Correlations between human experts and LLM judgments on generated problems. The charts illustrate a significant gap between human and LLM perceptions of quality and novelty (Finding 4), despite high inter-LLM agreement (bottom right). LLM-predicted difficulty shows a weak positive correlation with human-rated quality (top left), suggesting it can serve as a noisy proxy for quality estimation (Finding 5).