Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming

Sicheol Sung, Aditi, Dogyu kim, Yo-Sub Han, Sang-Ki Ko

TL;DR

This work tackles automated test case generation for competitive programming by translating natural-language problem specifications into Context-Free Grammars with Counters (CCFGs), enabling tight adherence to complex constraints. A specialized translation model, CCFGT5, maps NL descriptions to CCFGs using dual CodeT5 modules and a custom tokenizer, after which CCFGs generate valid, specification-compliant test cases. Evaluations on the CodeContests dataset show that CCFG-based generation, especially with CcfgT5, significantly improves set-based validity and fault-detection effectiveness while reducing reliance on handcrafted grammars. The approach offers a scalable, grammar-driven framework for robust competitive-programming evaluation and has potential applicability to broader automated testing tasks. Overall, the work demonstrates that integrating grammar-based formalism with neural translation can better capture complex input specifications and produce high-quality test cases.

Abstract

Automated Test Case Generation (ATCG) is crucial for evaluating software reliability, particularly in competitive programming where robust algorithm assessments depend on diverse and accurate test cases. However, existing ATCG methods often fail to meet complex specifications or generate effective corner cases, limiting their utility. In this work, we introduce Context-Free Grammars with Counters (CCFGs), a formalism that captures both syntactic and semantic structures in input specifications. Using a fine-tuned CodeT5 model, we translate natural language input specifications into CCFGs, enabling the systematic generation of high-quality test cases. Experiments on the CodeContests dataset demonstrate that CCFG-based test cases outperform baseline methods in identifying incorrect algorithms, achieving significant gains in validity and effectiveness. Our approach provides a scalable and reliable grammar-driven framework for enhancing automated competitive programming evaluations.

LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming

TL;DR

This work tackles automated test case generation for competitive programming by translating natural-language problem specifications into Context-Free Grammars with Counters (CCFGs), enabling tight adherence to complex constraints. A specialized translation model, CCFGT5, maps NL descriptions to CCFGs using dual CodeT5 modules and a custom tokenizer, after which CCFGs generate valid, specification-compliant test cases. Evaluations on the CodeContests dataset show that CCFG-based generation, especially with CcfgT5, significantly improves set-based validity and fault-detection effectiveness while reducing reliance on handcrafted grammars. The approach offers a scalable, grammar-driven framework for robust competitive-programming evaluation and has potential applicability to broader automated testing tasks. Overall, the work demonstrates that integrating grammar-based formalism with neural translation can better capture complex input specifications and produce high-quality test cases.

Abstract

Automated Test Case Generation (ATCG) is crucial for evaluating software reliability, particularly in competitive programming where robust algorithm assessments depend on diverse and accurate test cases. However, existing ATCG methods often fail to meet complex specifications or generate effective corner cases, limiting their utility. In this work, we introduce Context-Free Grammars with Counters (CCFGs), a formalism that captures both syntactic and semantic structures in input specifications. Using a fine-tuned CodeT5 model, we translate natural language input specifications into CCFGs, enabling the systematic generation of high-quality test cases. Experiments on the CodeContests dataset demonstrate that CCFG-based test cases outperform baseline methods in identifying incorrect algorithms, achieving significant gains in validity and effectiveness. Our approach provides a scalable and reliable grammar-driven framework for enhancing automated competitive programming evaluations.

Paper Structure

This paper contains 19 sections, 6 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Test Case Generation using LLMs
  4. Grammar-based Test Case Generation
  5. Methodology
  6. Context-Free Grammars With Counters
  7. CCFGT5 Translation Model
  8. Experiments
  9. Dataset
  10. Test Case Sampling
  11. Baselines
  12. Mutation-based fuzzing.
  13. Direct test case generation from LLMs.
  14. Evaluation Metrics
  15. Validity and generality.
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of the proposed framework for generating test cases for competitive programming problems. The deep learning model translates specifications into CCFGs while preserving their meaning. Subsequently, the CCFGs are utilized to generate test cases.

Theorems & Definitions (5)

  • Example 1: Input Specification
  • Example 2: Incorrect CFG of Example \ref{['example:specification']}
  • Example 3: Correct CCFG of Example \ref{['example:specification']}
  • Example 4: 139_D. Digit Permutations from Codeforces
  • Example 5: 103_C. Russian Roulette from Codeforces