Benchmarking Correctness and Security in Multi-Turn Code Generation

TL;DR

MT-Sec introduces a multi-turn benchmark for evaluating both correctness and security in coding tasks that reflect real-world iterative development. It synthesizes three-turn dialogues (expansion, editing, refactor) from seed single-turn secure-coding prompts and reuses the same dynamic correctness/security tests. Evaluated across 32 models and three agent scaffolds, MT-Sec demonstrates a consistent 20-27% drop in Correct & Secure outputs in multi-turn settings, with further degradation in code-diff tasks. The results highlight challenges in maintaining cross-turn coherence, the limited effectiveness of agent scaffolds in MT settings, and the need for secure, multi-turn benchmarks for safe deployment.

Abstract

AI coding assistants powered by large language models (LLMs) have transformed software development, significantly boosting productivity. While existing benchmarks evaluate the correctness and security of LLM-generated code, they are typically limited to single-turn tasks that do not reflect the iterative nature of real-world development. We introduce MT-Sec, the first benchmark to systematically evaluate both correctness and security in multi-turn coding scenarios. We construct this using a synthetic data pipeline that transforms existing single-turn tasks into semantically aligned multi-turn interaction sequences, allowing reuse of original test suites while modeling the complexity of real-world coding processes. We evaluate 32 open- and closed-source models, and three agent-scaffolding on MT-Sec and observe a consistent 20-27% drop in "correct and secure" outputs from single-turn to multi-turn settings -- even among state-of-the-art models. Beyond full-program generation, we also evaluate models on multi-turn code-diff generation -- an unexplored yet practically relevant setting -- and find that models perform worse here, with increased rates of functionally incorrect and insecure outputs. Finally, we find that while agent scaffoldings boost single-turn code generation performance, they are not quite as effective in multi-turn evaluations. Together, these findings highlight the need for benchmarks that jointly evaluate correctness and security in multi-turn, real-world coding workflows.

Figures (6)

• Figure 1: A comparison of single-turn coding to multi-turn scenarios, with three different interaction types. Our proposed dataset contains multi-turn conversations that are semantically aligned with their single-turn counterparts, sharing the same requirements. The same unit tests are applied to both to ensure a fair evaluation.
• Figure 2: MT-Sec is constructed in three stages: (i) selecting seed prompts from single-turn secure code benchmarks; (ii) synthetically converting them into multi-turn requests using a data-generator LLM with consistency guardrails; and (iii) manually verifying the validity of the multi-turn requests.
• Figure 3: Performance comparison between Single-Turn (ST), standard Multi-Turn (MT) settings, and a control condition, MT-Random on MT-SecCodePLT. In MT-Random, context length is matched to MT by including unrelated prior turns, isolating the effect of longer input without introducing cross-turn dependencies. Results across six models show that performance in MT-Random is comparable to, or slightly better than, ST--indicating that increased input length alone does not cause degradation.
• Figure 4:
• Figure 5: Comparison between Qwen3-8B (left) and its "thinking" variant (right) on the same multi-turn task. While the base model fails to enforce security constraints, the thinking model correctly distinguishes supported from secure algorithms and passes all tests. Thinking traces reveal explicit planning and code structuring, suggesting improved reasoning contributes to better security outcomes.