CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

TL;DR

CodeFlowBench addresses the lack of benchmarks for multi-turn, dependency-driven code generation by introducing a two-component suite (CodeFlowBench-Comp for algorithmic depth and CodeFlowBench-Repo for real-world software tasks) and a novel evaluation framework with structural metrics like APD and DSC. The authors present an automated data curation pipeline that decomposes monolithic solutions into subproblems via function dependencies and validates them with unit tests, enabling ground-truth codeflow tasks. Empirical results across 19 LLMs show substantial performance degradation in multi-turn settings, with depth of dependencies strongly impacting success and different model classes exhibiting distinct failure patterns (IR, IG, IM). The work suggests that future code generation systems must better handle long-range context, dependency management, and iterative refinement to be practical in real-world software development.

Abstract

Modern software development demands code that is maintainable, testable, and scalable by organizing the implementation into modular components with iterative reuse of existing codes. We formalize this iterative, multi-turn paradigm as codeflow and introduce CodeFlowBench, the first benchmark designed to comprehensively evaluate LLMs' ability to perform codeflow - implementing new functionality by reusing existing functions over multiple turns. CodeFlowBench comprises two complementary components: CodeFlowBench-Comp, a core collection of 5,000+ competitive programming problems from Codeforces updated via an automated pipeline and CodeFlowBench-Repo, which is sourced from GitHub repositories to better reflect real-world scenarios. Furthermore, a novel evaluation framework featured dual assessment protocol and structural metrics derived from dependency trees is introduced. Extensive experiments reveal significant performance degradation in multi-turn codeflow scenarios. Furthermore, our in-depth analysis illustrates that model performance inversely correlates with dependency complexity. These findings not only highlight the critical challenges for supporting real-world workflows, but also establish CodeFlowBench as an essential tool for advancing code generation research.

Figures (20)

• Figure 1: A currency arbitrage example contrasting the modular codeflow workflow (top) with a monolithic single-turn implementation (bottom).
• Figure 2: An illustrative example from CodeFlowBench (Source: Codeforces https://codeforces.com/problemset/problem/2055/E). The figure illustrates the problem definition with its dependency structure (top), the iterative subproblem decomposition (middle), and the corresponding LLM interaction workflow (bottom).
• Figure 3: The data curation pipeline of CodeFlowBench. In Data Preparation Phase (Steps I--III), we scrape problem statements and editorial information, followed by generating and validating the canonical solution codes. In Subproblem Generation Phase (Steps IV--VI), we decompose solutions into subproblems via AST parsing, generate corresponding test cases, and perform automatic complexity labeling to achieve the final CodeFlowBench.
• Figure 4: Models' Pass@1 results on multi-turn problems grouped by model categories and turn number.
• Figure 5: An example page of problems on Codeforces, which contains problem ID, title, time and memory complexity limits, problem description, input description, output description, sample tests, notes of each problem.The original problem is https://codeforces.com/problemset/problem/196/E