UniCode: A Framework for Generating High Quality Competitive Coding Problems

TL;DR

UniCode tackles the stagnation and contamination issues of static competitive programming benchmarks by introducing a generative evaluation framework that automatically creates high-quality problems and robust test suites. It combines evolution-inspired problem generation (single-problem extension, same-type fusion, cross-type fusion) with a stress-driven, multi-stage test-case pipeline that does not require a canonical ground-truth solution. The framework is validated on a 492-problem benchmark tested against 19 LLMs, revealing a challenging and highly discriminative landscape where even strong models struggle on hard cases. The work demonstrates solid reproducibility and meaningful diversity, with open release artifacts and insights into model generalization and the trade-offs between reasoning capacity and template-driven tasks, offering a scalable path for continuous evaluation in the coding domain.

Abstract

The reliance of competitive coding benchmarks on static, human-authored problems creates significant challenges, including data contamination and limited scalability. To address these issues, we introduce UniCode, a novel framework that automatically generates high-quality algorithmic problems alongside robust, contamination-resistant test cases. Inspired by biological evolution that creates better and diverse offspring, our framework leverages Large Language Models (LLMs) to systematically diversify problems through three strategies: single problem extension, same-type fusion, and cross-type fusion. A key innovation is our stress-driven test case synthesis pipeline, which generates reliable test suites without requiring a canonical ground-truth solution. This pipeline combines brute-force grounding for small-scale inputs with a consensus-based validation mechanism for large-scale inputs to ensure high correctness and coverage. We demonstrate effectiveness of our framework by curating a benchmark of 492 problems and evaluating 19 state-of-the-art LLMs. The results reveal that UniCode is highly challenging and discriminative, with the top-performing model, o4-mini, achieving a pass rate of only 70.3%. Our framework provides a scalable and reliable solution for generating dynamic evaluation datasets in coding domain.

Figures (9)

• Figure 1: An illustration of our three problem generation strategies using seed problems. Same-type fusion (left): combines two problems sharing a common tag (e.g., hash table) to produce a new problem with the same core logic but greatly different descriptions. Single-problem extension (middle): modifies a seed problem to create a more complex version (e.g., Two Sum$\rightarrow$Three sum). Cross-type fusion (right): merges problems from different categories to create a hybrid challenge (e.g., Palindrome Sum Pair). For concrete examples of generated problems, please refer to Appendix \ref{['app:example_problems']}.
• Figure 2: Stress-driven pipeline for ground-truth generation. (1) Stress Testing: A brute-force solver establishes trusted outputs for small-scale inputs. These are used as a "stress test" to filter a pool of LLM-generated optimized solutions. (2) Consensus Validation: The filtered, efficient solvers are run on large-scale inputs, and outputs are determined by a strict majority vote. (3) LLM Adjudication: A powerful LLM adjudicates between conflicting outputs for inputs where no majority is reached.
• Figure 3: Performance comparison across question sets. SeedQS denotes the original seed questions, ShadowQS represents the semantically equivalent shadow questions generated from SeedQS, and CodeGenQS refers to the novel question set created by our UniCode method.
• Figure 4: Comparative performance across problem types for closed-source (left) and open-source (right) models. The x-axis represents algorithmic tags. The y-axis refers to the pass@1 rate. Best viewed zoomed in.
• Figure 5: Unicode leaderboard. Pass@1 score vs. average cost per problem across various models.