CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming

TL;DR

This work addresses the problem of duplicate and highly similar problems in competitive programming benchmarks by introducing CPRet-PCPCD, a retrieval-focused benchmark with four tasks (two code-centric, two problem-centric) and temporally separated test sets. It develops two task-specialized retrievers, CPRetriever-Code and CPRetriever-Prob, employing Group-InfoNCE to align problems with multiple correct solutions and to tailor retrieval for problem-level similarity, respectively. Empirical results show strong performance of the code-focused model on code tasks and the problem-focused model on problem tasks, while also revealing that high problem similarity inflates pass rates and reduces model differentiation, underscoring the need for similarity-aware evaluation. The work provides practical retrieval tools and insights for fair benchmarking, with implications for retrieval-augmented generation and contest integrity in CP ecosystems.

Abstract

Competitive programming benchmarks are widely used in scenarios such as programming contests and large language model assessments. However, the growing presence of duplicate or highly similar problems raises concerns not only about competition fairness, but also about the validity of competitive programming as a benchmark for model evaluation. In this paper, we propose a new problem, similar question retrieval, to tackle this issue. Due to the lack of both data and models, solving this problem is challenging. To this end, we introduce CPRet, a retrieval-oriented benchmark suite for competitive programming, covering four retrieval tasks: two code-centric (i.e., Text-to-Code, Code-to-Code) and two newly proposed problem-centric tasks (i.e., Problem-to-Duplicate, Simplified-to-Full) built from a combination of automatically crawled problem-solution data and manually curated annotations. Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation. Besides, we further develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem-code alignment, and CPRetriever-Prob, fine-tuned for identifying problem-level similarity. Both models achieve strong results and are open-sourced for local use. Finally, we analyze LiveCodeBench and find that high-similarity problems inflate model pass rates and reduce differentiation, underscoring the need for similarity-aware evaluation in future benchmarks. Github: https://github.com/coldchair/CPRet Online Demo: https://www.cpret.online/

Figures (10)

• Figure 1: Year-wise trend of annotated duplicate problem discussions.
• Figure 2: Left: NDCG@10 scores on CoIR/APPS across model release dates, showing steady performance improvement as models evolve. Right: Performance on the Text-to-Code task grouped by problem release year, revealing sharp degradation on older problems, especially around major dataset release points (Description2Code, APPS, CodeContests).
• Figure 3: Impact of similarity to prior problems on pass rate.Left: For 388 post-2024.9.1 LiveCodeBench problems, we plot average model pass rate vs. maximum similarity to historical CPRet problems, computed via CPRetriever-Prob. Regression lines are shown by difficulty, with the gray line indicating the overall trend across all problems. Right: Pass rates increase monotonically across similarity bins, confirming a strong link between retrieval similarity and generation success.
• Figure 4: Left: Models with higher capacity (High) maintain relatively high performance even on low-similarity problems, while lower-capacity models benefit more from high-similarity problems. Right: Easy problems show slightly higher maximum similarity to past problems than hard ones, but the difference is limited.
• Figure 5: Dataset-level characteristics by language and format in CPRet-PCPCD.